Apparatus and method for conditional decoder-side motion vector refinement in video coding

ABSTRACT

A method for inter-prediction of a current image block in a current picture of a video is provided. The method includes determining whether a first temporal distance (such as TD0) is equal to a second temporal distance (such as TD1), wherein the first temporal distance is represented in terms of a difference between a picture order count value of the current picture and a picture order count value of a first reference image; and the second temporal distance is represented in terms of a difference between a picture order count value of a second reference image and the picture order count value of the current picture; and performing no motion vector refinement (DMVR) procedure when it is determined that the first temporal distance (TD0) is not equal to the second temporal distance (TD1). Thus the DMVR procedure is restricted to only the image block with equal-distance references.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2019/105717, filed on Sep. 12, 2019, which claims priority toIndia Provisional Patent Application No. IN201831034607, filed on Sep.13, 2018. The disclosures of the aforementioned applications are herebyincorporated by reference in their entireties.

TECHNICAL FIELD

Embodiments of the present disclosure are generally related to videodata encoding and decoding techniques, and are especially related todecoder-side motion vector refinement (DMVR) in video coding.

BACKGROUND

The amount of video data needed to depict even a relatively short videocan be substantial, which may result in difficulties when the data is tobe streamed or otherwise communicated across a communications networkwith limited bandwidth capacity. Thus, video data is generallycompressed before being communicated across modern daytelecommunications networks. The size of a video could also be an issuewhen the video is stored on a storage device because memory resourcesmay be limited. Video compression devices often use software and/orhardware at the source to code the video data prior to transmission orstorage, thereby decreasing the quantity of data needed to representdigital video images. The compressed data is then received at thedestination by a video decompression device that decodes the video data.With limited network resources and ever increasing demands of highervideo quality, improved compression and decompression techniques thatimprove compression ratio with little to no sacrifice in image qualityare desirable.

SUMMARY

Embodiments of the present application provide apparatus and method forinter-prediction of a current image block in a current picture of avideo, an encoder and a decoder that can perform decoder side motionvector refinement (DMVR) conditionally, and thus the coding efficiencycan be improved.

Embodiments of the invention are defined by the features of theindependent claims, and further advantageous implementations of theembodiments by the features of the dependent claims.

Particular embodiments are outlined in the attached independent claims,with other embodiments in the dependent claims.

According to a first aspect, the disclosure relates to a method forinter-prediction (bi-prediction) of a current image block in a currentpicture of a video, the method comprising: determining whether thecurrent picture is temporally (with regard to time) between a firstreference picture (such as RefPic0) and a second reference picture (suchas RefPic1) and whether a first temporal distance (such as TD0) and asecond temporal distance (such as TD1) are the same distance, whereinthe first temporal distance (TD0) is between the current picture and thefirst reference picture (RefPic0), and the second temporal distance(TD1) is between the current picture and the second reference picture(RefPic1); and performing motion vector refinement (DMVR) procedure toobtain a position of a first refined reference block and a position of asecond refined reference block and determining a prediction block(predicted sample values) of the current image block based on theposition of the first refined reference block and the position of thesecond refined reference block, when it is determined that the currentpicture is temporally between the first reference picture (such asRefPic0) and the second reference picture (such as RefPic1) and that thefirst temporal distance (TD0) and the second temporal distance (TD1) arethe same distance.

It is noted that “when it is determined that the current picture istemporally between the first reference picture (such as RefPic0) and thesecond reference picture (such as RefPic1) and that the first temporaldistance (TD0) and the second temporal distance (TD1) are the samedistance” should not be understood as “only when it is determined thatthe current picture is temporally between the first reference picture(such as RefPic0) and the second reference picture (such as RefPic1) andthat the first temporal distance (TD0) and the second temporal distance(TD1) are the same distance”. Other conditions can be also consideredwhen determining whether to perform motion vector refinement (DMVR)procedure.

Regarding “a position of a first refined reference block and a positionof a second refined reference block”, the position can be an absoluteposition, which is the position in the reference picture, or a relativeposition which is a position offset based on the position of the initialreference block.

It is noted that DMVR is applied for the merge mode of bi-predictionwith one motion vector (MV1) from a reference picture in the past andanother MV from another reference picture in the future. The referencepictures may be two pictures in temporally different directions withrespect to the current picture that contains the current image block.The present disclosure is not applicable to the scenario in which bothpredictions come from the same time direction (either both from the pastor both from the future).

In a possible implementation form of the method according to anypreceding implementation of the first aspect, the determining whetherthe current picture is temporally between a first reference picture(such as RefPic0) and a second reference picture (such as RefPic1) andthat a first temporal distance (such as TD0) and a second temporaldistance (such as TD1) is the same distance, wherein the first temporaldistance (TD0) is between the current picture and the first referencepicture (RefPic0), and the second temporal distance (TD1) is between thecurrent picture and the second reference picture (RefPic1), comprises:

|TD0|==|TD1|

AND

TD0*TD1<0

For each merge candidate which indicates bi-direction, compute TD0 andTD1 as temporal distances of L0 and L1 reference picture from thecurrent picture. The TD0 and TD1 may be calculated by using a pictureorder count (POC). For example:

TD0=POCc−POC ₀

TD1=POCc−POC ₁

Here, POCc, POC₀, and POC₁ represent POC of the current picture, POC ofthe first reference picture, and POC of the second reference picture,respectively.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, the method further comprises: performing motion compensation usinga first initial motion vector (MV0) and a second initial motion vector(MV1), when it is determined that the first temporal distance (TD0) andthe second temporal distance (TD1) are different distance or that thecurrent picture is not temporally between the first reference picture(such as RefPic0) and the second reference picture (such as RefPic1); inone example, performing motion compensation using a first initial motionvector (MV0) and a second initial motion vector (MV1) when |TD0|≠|TD1|or TD0*TD1>=0 in the case that TD0=POCc−POC₀, and TD1=POCc−POC₁.Alternatively, in another example, performing motion compensation usinga first initial motion vector (MV0) and a second initial motion vector(MV1), when TD0≠TD1 in the case that TD0=POCc−POC₀, and TD1=POC₁−POCc.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, the method further comprising: obtaining an initial motioninformation of the current image block in the current picture, whereinthe initial motion information comprises the first initial motionvector, a first reference index, the second initial motion vector and asecond reference index, wherein the first reference index indicates thefirst reference picture, and the second reference index indicates thesecond reference picture.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, the step of performing motion vector refinement (DMVR) procedure,comprises:

determining a first initial reference block in a first reference picturebased on the first initial motion vector;determining a second initial reference block in a second referencepicture based on the second initial motion vector;generating a bilateral reference block based on the first initialreference block and the second initial reference block; for example, thebilateral reference block may be referred as bilateral template, and thetemplate has the shape and size of the image prediction block;comparing a cost between the bilateral reference block and each of aplurality of first reference blocks in the first reference picture todetermine a position of a first refined reference block or a firstrefined motion vector; andcomparing a cost between the bilateral reference block and each of aplurality of second reference blocks in the second reference picture todetermine a position of a second refined reference block or a secondrefined motion vector.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, the step of performing motion vector refinement (DMVR) procedure,comprises:

obtaining a template for the current image block based on a firstinitial reference block pointed to by the first initial motion vector inthe first reference picture (such as RefPic0) and a second initialreference block pointed to by the second initial motion vector in thesecond reference picture (such as RefPic1); anddetermining the first refined motion vector and the second refinedmotion vector by template matching with said template in a first searchspace and a second search space respectively, the first search spacecontaining a position given by the first initial motion vector and thesecond search space containing a position given by the second initialmotion vector.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, the step of performing motion vector refinement (DMVR) procedure,comprises:

determining a pair of best-matching blocks pointed to by best motionvectors from a plurality of pairs of reference blocks based on thematching cost criterion (for example, based on the matching cost of eachpair of reference blocks), wherein said pair of reference blocksincludes a first reference block in a sample region that is of the firstreference picture and that is determined based on the first initialmotion vector and a second reference block in a sample region that is ofthe second reference picture and that is determined based on the secondinitial motion vector;wherein the best motion vectors include the first refined motion vectorand the second refined motion vector.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, the step of performing motion vector refinement (DMVR) procedureto obtain a position of a first refined reference block and a positionof a second refined reference block, comprises:

determining positions of N first reference blocks and positions of Nsecond reference blocks based on the first initial motion vector, thesecond initial motion vector and a position of the current image block,wherein the N first reference blocks are included in the first referenceimage, and the N second reference blocks are included in the secondreference image, and N is an integer greater than 1; anddetermining positions of a pair of reference blocks from the positionsof the M pairs of reference blocks as a position of the first refinedreference block and a position of the second refined reference blockbased on the matching cost criterion, wherein positions of each pair ofreference blocks include a position of a first reference block and aposition of a second reference block, and for each pair of referenceblocks, the first position offset (delta0x, delta0y) and the secondposition offset (delta1x, delta1y) are mirrored, and the first positionoffset (delta0x, delta0y) represents an offset of the position of thefirst reference block relative to the position of the first initialreference block, and the second position offset (delta1x, delta1y)represents an offset of the position of the second reference blockrelative to the position of the second initial reference block, whereinM is an integer greater than or equal to 1, and M is less than or equalto N.

In an example, the expression that the first position offset (delta0x,delta0y) and the second position offset (delta1x, delta1y) are mirrored,may be understood as: the direction of the first position offset isopposite to the direction of the second position offset, and themagnitude of the first position offset is the same as the magnitude ofthe second position offset.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, wherein the step of determining a prediction block of the currentimage block based on the position of the first refined reference blockand the position of the second refined reference block, comprises:

determining a prediction block based on the first refined referenceblock and the second refined reference block, wherein the first refinedreference block is determined in the first reference picture based onthe position of the first refined reference block and the second refinedreference block is determined in the second reference picture based onthe position of the second refined reference block; ordetermining a prediction block based on the first refined referenceblock and a second refined reference block, wherein the first refinedreference block and the second refined reference block are determined byperforming motion compensation using the first refined motion vector andthe second refined motion vector.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, wherein the first reference picture is a reference imagetemporally preceding the current picture and the second referencepicture is a reference image preceded temporally by the current picture;or the first reference picture is a reference image preceded temporallyby the current picture and the second reference picture is a referenceimage temporally preceding the current picture; or

wherein the first reference picture is a reference image in the past andthe second reference picture is a reference image in the future; or thefirst reference picture is a reference image in the future and thesecond reference picture is a reference image in the past.In other words, a previous picture of the current picture is the firstreference picture, and a next image of the current picture is the secondreference picture; or a previous picture of the current picture is thesecond reference picture, and a next image of the current picture is thefirst reference picture.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, wherein the first temporal distance (TD0) indicates the pictureorder count (POC) distance between the current picture and the firstreference picture, and the second temporal distance (TD1) indicates thePOC distance between the current picture and the second referencepicture; or

wherein the first temporal distance (TD0) is represented in terms ofdifference between the picture order count value (POCc) of the currentpicture and the picture order count value (POC0) of the first referenceimage; and the second temporal distance (TD1) is represented in terms ofdifference between the picture order count value (POCc) of the currentpicture and the picture order count value (POC1) of the second referenceimage.

In a possible implementation form of the method according to anypreceding implementation of the first aspect or the first aspect assuch, wherein the step of determining whether the current picture istemporally between a first reference picture (such as RefPic0) and asecond reference picture (such as RefPic1), comprises:

determining whether the picture order count value (POCc) of the currentpicture is larger than the picture order count value (POC0) of the firstreference image and picture order count value (POCc) of the currentpicture is smaller than the picture order count value (POC1) of thesecond reference image, or whether the picture order count value (POCc)of the current picture is smaller than the picture order count value(POC0) of the first reference image and picture order count value (POCc)of the current picture is larger than the picture order count value(POC1) of the second reference image.

According to a second aspect, the disclosure relates to a method forinter-prediction (bi-prediction) of a current image block in a currentpicture of a video, the method comprising: determining whether thecurrent picture is temporally between a first reference picture (such asRefPic0) and a second reference picture (such as RefPic1) and whether afirst temporal distance (such as TD0) and a second temporal distance(such as TD1) are the same distance, wherein the first temporal distance(TD0) is between the current picture and the first reference picture(RefPic0), and the second temporal distance (TD1) is between the currentpicture and the second reference picture (RefPic1); and performingmotion compensation using a first initial motion vector (MV0) and asecond initial motion vector (MV1), when it is determined that the firsttemporal distance (TD0) and the second temporal distance (TD1) aredifferent distance or that the current picture is not temporally betweenthe first reference picture (such as RefPic0) and the second referencepicture (such as RefPic1).

In a possible implementation form of the method according to the secondaspect as such, wherein initial motion information of the current imageblock comprises the first initial motion vector, a first referenceindex, the second initial motion vector and a second reference index,wherein the first reference index indicates the first reference picture,and the second reference index indicates the second reference picture.

In a possible implementation form of the method according to anypreceding implementation of the second aspect or the second aspect assuch, wherein the first temporal distance indicates a picture ordercount (POC) distance between the current picture and the first referencepicture, and the second temporal distance indicates a POC distancebetween the current picture and the second reference picture; or

wherein the first temporal distance (TD0) is represented in terms ofdifference between a picture order count value (POCc) of the currentpicture and a picture order count value (POC0) of the first referenceimage; and the second temporal distance (TD1) is represented in terms ofdifference between the picture order count value (POCc) of the currentpicture and a picture order count value (POC1) of the second referenceimage.

According to a third aspect, the disclosure relates to a method forinter-prediction (bi-prediction) of a current image block in a currentpicture of a video, the method comprising:

determining whether a first temporal distance (such as TD0) is equal toa second temporal distance (such as TD1), wherein the first temporaldistance (TD0) is represented in terms of difference between the pictureorder count value (POCc) of the current picture and the picture ordercount value (POC0) of the first reference image; and the second temporaldistance (TD1) is represented in terms of difference between the pictureorder count value (POC1) of the second reference image and the pictureorder count value (POCc) of the current picture; and performing motionvector refinement (DMVR) procedure to determine a prediction block ofthe current image block, when it is determined that the first temporaldistance (TD0) is equal to the second temporal distance (TD1).

In a possible implementation form of the method according to the thirdaspect as such, the method further comprises: performing motioncompensation using a first initial motion vector and a second initialmotion vector to determine a prediction block of the current imageblock, when it is determined that the first temporal distance is notequal to the second temporal distance.

According to a fourth aspect, the disclosure relates to a method forinter-prediction (bi-prediction) of a current image block in a currentpicture of a video, the method comprising:

determining whether a first temporal distance (such as TD0) is equal toa second temporal distance (such as TD1), wherein the first temporaldistance (TD0) is represented in terms of difference between the pictureorder count value (POCc) of the current picture and the picture ordercount value (POC0) of the first reference image; and the second temporaldistance (TD1) is represented in terms of difference between the pictureorder count value (POC1) of the second reference image and the pictureorder count value (POCc) of the current picture; andperforming no motion vector refinement (DMVR) procedure (or disablingmotion vector refinement (DMVR) procedure), when it is determined thatthe first temporal distance (TD0) is not equal to the second temporaldistance (TD1).

For each merge candidate which indicates bi-direction, the TD0 and TD1may be calculated by using a picture order count (POC). For example:

TD0=POCc−POC ₀

TD1=POC ₁ −POCc

Here, POCc, POC₀, and POC₁ represent POC of the current picture, POC ofthe first reference picture, and POC of the second reference picturerespectively.

In a possible implementation form of the method according to anypreceding implementation of the second or third or fourth aspect or thesecond or third or fourth aspect as such, wherein the first referencepicture is a reference image temporally preceding the current pictureand the second reference picture is a reference image precededtemporally by the current picture; or the first reference picture is areference image preceded temporally by the current picture and thesecond reference picture is a reference image temporally preceding thecurrent picture; or wherein the first reference picture is a referenceimage in the past and the second reference picture is a reference imagein the future; or the first reference picture is a reference image inthe future and the second reference picture is a reference image in thepast.

In other words, a previous picture of the current picture is the firstreference picture, and a next image of the current picture is the secondreference picture; or a previous picture of the current picture is thesecond reference picture, and a next image of the current picture is thefirst reference picture.

According to a fifth aspect, the disclosure relates to a method forencoding a video image comprising:

performing inter-prediction of a current image block in a currentpicture of the video to obtain a prediction block of the current imageblock according to the above method; andencoding difference (e.g. residual or residual block) between thecurrent image block and the prediction block and generating a bitstreamincluding the encoded difference and an index (such as a merge candidateindex) for indicating an initial motion information, wherein the initialmotion information comprises a first initial motion vector and a secondinitial motion vector.

According to a sixth aspect, the disclosure relates to a method fordecoding a video image from a bitstream comprising:

parsing from the bitstream an index (such as a merge candidate index)for indicating an initial motion information and an encoded difference(e.g. residual or residual block) between a current image block and aprediction block of the current image block, wherein the initial motioninformation comprises a first initial motion vector and a second initialmotion vector;performing inter-prediction of a current image block in a currentpicture of the video to obtain a prediction block of the current imageblock according to the method previously described above; andreconstructing the current image block as a sum of the parsed differenceand the prediction block.

According to a seventh aspect, the disclosure relates to a method ofencoding implemented by an encoding device, comprising:

determining value of a syntax element indicating whether the abovemethod is enabled or not; and generating a bitstream including thesyntax element.

In a possible implementation form of the method according to anypreceding implementation of the seventh aspect or the seventh aspect assuch, wherein the syntax element is signaled at any one of a sequenceparameter set (SPS) level, a picture parameter set (PPS) level, a sliceheader, coding tree unit (CTU) syntax, or coding unit (CU) syntax.

In a possible implementation form of the method according to anypreceding implementation of the seventh aspect or the seventh aspect assuch, wherein the syntax element comprises a first flag (such assps_conditional_dmvr_flag) and/or a second flag (such aspps_conditional_dmvr_flag);

If the first flag (sps_conditional_dmvr_flag) is equal to 0, the methodpreviously described above is not performed for the image blocks of thesequence;If the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 0, the methodpreviously described above is not performed for the image blocks of thepicture of the sequence; orIf the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 1, the methodpreviously described above is performed for the image blocks of thepicture of the sequence.

According to an eighth aspect, the disclosure relates a method ofdecoding implemented by a decoding device, comprising:

parsing from a bitstream a syntax element indicating whether thepreviously described above method is enabled or not; andadaptively enabling or disabling decoder-side motion vector refinement(DMVR) procedure according to the syntax element indicating whether theabove method is enabled or not.

In a possible implementation form of the method according to anypreceding implementation of the eighth aspect or the eighth aspect assuch, wherein the syntax element is obtained from any one of a sequenceparameter set (SPS) level of a bitstream, a picture parameter set (PPS)level of the bitstream, a slice header, coding tree unit (CTU) syntax,or coding unit (CU) syntax.

In a possible implementation form of the method according to anypreceding implementation of the eighth aspect or the eighth aspect assuch, wherein the syntax element comprises the first flag(sps_conditional_dmvr_flag) and the second flag(pps_conditional_dmvr_flag);

If the first flag (sps_conditional_dmvr_flag) is equal to 0, the methodpreviously described above is not performed for the image blocks of thesequence;If the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 0, the methodpreviously described above is not performed for the image blocks of thepicture of the sequence; orIf the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 1, the methodpreviously described above is performed for the image blocks of thepicture of the sequence.

According to a ninth aspect, the disclosure relates a coding device,comprising:

a memory storing instructions; anda processor coupled to the memory, the processor configured to executethe instructions stored in the memory to cause the processor to performthe previously shown method.

According to a tenth aspect, the disclosure relates an apparatus forinter-prediction of a current image block in a current picture of avideo, comprising:

a determining unit configured to determine whether the current pictureis temporally between a first reference picture (such as RefPic0) and asecond reference picture (such as RefPic1) and whether a first temporaldistance (such as TD0) and a second temporal distance (such as TD1) arethe same distance, wherein the first temporal distance (TD0) is betweenthe current picture and the first reference picture (RefPic0), and thesecond temporal distance (TD1) is between the current picture and thesecond reference picture (RefPic1); andan inter prediction processing unit configured to perform motion vectorrefinement (DMVR) procedure to obtain a position of a first refinedreference block and a position of a second refined reference block andto determine a prediction block of the current image block based on theposition of the first refined reference block and the position of thesecond refined reference block, when it is determined that the currentpicture is temporally between the first reference picture (such asRefPic0) and the second reference picture (such as RefPic1) and that thefirst temporal distance (TD0) and the second temporal distance (TD1) arethe same distance.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect, the determining unit isconfigured to determine whether the current picture is temporallybetween a first reference picture (such as RefPic0) and a secondreference picture (such as RefPic1) and whether a first temporaldistance (such as TD0) and a second temporal distance (such as TD1) arethe same distance through the following equation:

|TD0|==|TD1|

AND

TD0*TD1<0

For each merge candidate which indicates bi-direction, compute TD0 andTD1 as temporal distances of L0 and L1 reference picture from thecurrent picture. The TD0 and TD1 may be calculated by using a pictureorder count (POC). For example:

TD0=POCc−POC ₀

TD1=POCc−POC ₁

Here, POCc, POC₀, and POC₁ represent POC of the current picture, POC ofthe first reference picture, and POC of the second reference picture,respectively.

In a possible implementation form of the method according to the tenthaspect as such, wherein the inter prediction processing unit is furtherconfigured to perform motion compensation using a first initial motionvector (MV0) and a second initial motion vector (MV1), when it isdetermined that the first temporal distance (TD0) and the secondtemporal distance (TD1) are different distance or that the currentpicture is not temporally between the first reference picture (such asRefPic0) and the second reference picture (such as RefPic1). In oneexample, the inter prediction processing unit is further configured toperform motion compensation using a first initial motion vector (MV0)and a second initial motion vector (MV1), when |TD0|≠|TD1 or TD0*TD1>=0in the case that TD0=POCc−POC₀, and TD1=POCc−POC₁. Alternatively, inanother example, the inter prediction processing unit is furtherconfigured to perform motion compensation using a first initial motionvector (MV0) and a second initial motion vector (MV1), when TD0≠TD1 inthe case that TD0=POCc−POC₀, and TD1=POC₁−POCc.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, wherein the inter prediction processing unit is further configuredto

obtain an initial motion information of the current image block in thecurrent picture, wherein the initial motion information comprises thefirst initial motion vector, a first reference index, the second initialmotion vector and a second reference index, wherein the first referenceindex indicates the first reference picture, and the second referenceindex indicates the second reference picture.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, for the performing motion vector refinement (DMVR) procedure toobtain a position of a first refined reference block and a position of asecond refined reference block, wherein the inter prediction processingunit is specifically configured to:

determine a first initial reference block in a first reference picturebased on the first initial motion vector;determine a second initial reference block in a second reference picturebased on the second initial motion vector;generate a bilateral reference block based on the first initialreference block and the second initial reference block;compare a cost between the bilateral reference block and each of aplurality of first reference blocks in the first reference picture todetermine a position of a first refined reference block or a firstrefined motion vector;compare a cost between the bilateral reference block and each of aplurality of second reference blocks in the second reference picture todetermine a position of a second refined reference block or a secondrefined motion vector.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, for performing motion vector refinement (DMVR) procedure to obtaina position of a first refined reference block and a position of a secondrefined reference block, wherein the inter prediction processing unit isspecifically configured to:

obtain a template for the current image block based on a first initialreference block pointed to by the first initial motion vector in thefirst reference picture (such as RefPic0) and a second initial referenceblock pointed to by the second initial motion vector in the secondreference picture (such as RefPic1); anddetermine the first refined motion vector and the second refined motionvector by template matching with said template in a first search spaceand a second search space respectively, wherein the first search spaceis located on a position given by the first initial motion vector andthe second search space is located on a position given by the secondinitial motion vector.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, for the performing motion vector refinement (DMVR) procedure toobtain a position of a first refined reference block and a position of asecond refined reference block, wherein the inter prediction processingunit is specifically configured to:

determine a pair of best-matching blocks pointed to by best motionvectors from a plurality of pairs of reference blocks based on thematching cost criterion (for example, based on the matching cost of eachpair of reference blocks), wherein said pair of reference blocksincludes a first reference block in a sample region that is of the firstreference picture and that is determined based on the first initialmotion vector and a second reference block in a sample region that is ofthe second reference picture and that is determined based on the secondinitial motion vector;wherein the best motion vectors include the first refined motion vectorand the second refined motion vector.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, for the performing motion vector refinement (DMVR) procedure toobtain a position of a first refined reference block and a position of asecond refined reference block, wherein the inter prediction processingunit is specifically configured to: determine positions of N firstreference blocks and positions of N second reference blocks based on thefirst initial motion vector, the second initial motion vector and aposition of the current image block, wherein the N first referenceblocks are included in the first reference image, and the N secondreference blocks are included in the second reference image, and N is aninteger greater than 1; and determine positions of a pair of referenceblocks from the positions of the M pairs of reference blocks as aposition of the first refined reference block and a position of thesecond refined reference block based on the matching cost criterion,wherein positions of each pair of reference blocks includes a positionof a first reference block and a position of a second reference block,and for each pair of reference blocks, the first position offset(delta0x, delta0y) and the second position offset (delta1x, delta1y) aremirrored, and the first position offset (delta0x, delta0y) represents anoffset of the position of the first reference block relative to theposition of the first initial reference block, and the second positionoffset (delta1x, delta1y) represents an offset of the position of thesecond reference block relative to the position of the second initialreference block, wherein M is an integer greater than or equal to 1, andM is less than or equal to N.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, wherein for the determining a prediction block of the currentimage block based on the position of the first refined reference blockand the position of the second refined reference block, wherein theinter prediction processing unit is specifically configured to:

determine a prediction block based on the first refined reference blockand the second refined reference block, wherein the first refinedreference block is determined in the first reference picture based onthe position of the first refined reference block and the second refinedreference block is determined in the second reference picture based onthe position of the second refined reference block; ordetermine a prediction block based on the first refined reference blockand a second refined reference block, wherein the first refinedreference block and the second refined reference block are determined byperforming motion compensation using the first refined motion vector andthe second refined motion vector.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, wherein the first reference picture is a reference imagetemporally preceding the current picture and the second referencepicture is a reference image preceded temporally by the current picture;or the first reference picture is a reference image preceded temporallyby the current picture and the second reference picture is a referenceimage temporally preceding the current picture; or

wherein the first reference picture is a reference image in the past andthe second reference picture is a reference image in the future; or thefirst reference picture is a reference image in the future and thesecond reference picture is a reference image in the past.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, wherein the first temporal distance (TD0) indicates the pictureorder count (POC) distance between the current picture and the firstreference picture, and the second temporal distance (TD1) indicates thePOC distance between the current picture and the second referencepicture; or

wherein the first temporal distance (TD0) is represented in terms ofdifference between the picture order count value (POCc) of the currentpicture and the picture order count value (POC0) of the first referenceimage; and the second temporal distance (TD1) is represented in terms ofdifference between the picture order count value (POCc) of the currentpicture and the picture order count value (POC1) of the second referenceimage.

In a possible implementation form of the method according to anypreceding implementation of the tenth aspect or the tenth aspect assuch, wherein for the determining whether the current picture istemporally between a first reference picture (such as RefPic0) and asecond reference picture (such as RefPic1), wherein the inter predictionprocessing unit is specifically configured to:

determine whether the picture order count value (POCc) of the currentpicture is larger than the picture order count value (POC0) of the firstreference image and picture order count value (POCc) of the currentpicture is smaller than the picture order count value (POC1) of thesecond reference image, or whether the picture order count value (POCc)of the current picture is smaller than the picture order count value(POC0) of the first reference image and picture order count value (POCc)of the current picture is larger than the picture order count value(POC1) of the second reference image.

According to an eleventh aspect, the disclosure relates an encodingapparatus for encoding a video image, the encoding apparatus comprising:

a determining unit configured to determine whether the current pictureis temporally between a first reference picture and a second referencepicture and whether a first temporal distance and a second temporaldistance are the same distance, wherein the first temporal distance isbetween the current picture and the first reference picture, and thesecond temporal distance is between the current picture and the secondreference picture; andan inter prediction processing unit configured to perform motioncompensation using a first initial motion vector and a second initialmotion vector to determine a prediction block of the current imageblock, when it is determined that the first temporal distance and thesecond temporal distance are different distance or that the currentpicture is not temporally between the first reference picture and thesecond reference picture.

In a possible implementation form of the method according to theeleventh aspect as such, wherein initial motion information of thecurrent image block comprises the first initial motion vector, a firstreference index, the second initial motion vector and a second referenceindex, wherein the first reference index indicates the first referencepicture, and the second reference index indicates the second referencepicture.

In a possible implementation form of the method according to anypreceding implementation of the eleventh aspect or the eleventh aspectas such, wherein the first temporal distance indicates a picture ordercount (POC) distance between the current picture and the first referencepicture, and the second temporal distance indicates a POC distancebetween the current picture and the second reference picture; or whereinthe first temporal distance (TD0) is represented in terms of differencebetween a picture order count value (POCc) of the current picture and apicture order count value (POC0) of the first reference image; and thesecond temporal distance (TD1) is represented in terms of differencebetween the picture order count value (POCc) of the current picture anda picture order count value (POC1) of the second reference image.

According to a twelfth aspect, the disclosure relates an encodingapparatus for encoding a video image, the encoding apparatus comprising:

the previously shown apparatus for obtaining a prediction block of acurrent image block,an entropy coding unit configured to encoding difference (such asresidual) between the current image block and the prediction block ofthe current image block and generating a bitstream including the encodeddifference and an index (such as merge candidate index) for indicatinginitial motion information, wherein the initial motion informationcomprises a first initial motion vector and a second initial motionvector.

According to a thirteenth aspect, the disclosure relates a decodingapparatus for decoding a video image from a bitstream, the apparatuscomprising:

an entropy decoding unit configured to parse from the bitstream an indexfor indicating an initial motion information and an encoded difference(such as residual) between a current image block and a prediction blockof the current image block, wherein the initial motion informationcomprises a first initial motion vector and a second initial motionvector;the previously shown apparatus for obtaining the prediction block of thecurrent image block, andan image reconstruction unit configured to reconstruct the current imageblock as a sum of the parsed difference (such as residual) and theprediction block.

According to a fourteenth aspect, the disclosure relates an encodingdevice, comprising one or more processing circuitry configured to:

determine a value of a syntax element indicating whether the previouslyshown method is enable or not; and generate a bitstream including thesyntax element.

In a possible implementation form of the method according to thefourteenth aspect as such, wherein the syntax element is signaled at anyone of a sequence parameter set (SPS) level, a picture parameter set(PPS) level, a slice header, coding tree unit (CTU) syntax, or codingunit (CU) syntax.

In a possible implementation form of the method according to anypreceding implementation of the fourteenth aspect or the fourteenthaspect as such, wherein the syntax element comprises the first flag(sps_conditional_dmvr_flag) and the second flag(pps_conditional_dmvr_flag);

if the first flag (sps_conditional_dmvr_flag) is equal to 0, thepreviously shown method is not performed for the image blocks of thesequence;if the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 0, the previouslyshown method is not performed for the image blocks of the picture of thesequence; orif the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 1, the previouslyshown method 1 is performed for the image blocks of the picture of thesequence.

According to a fifteenth aspect, the disclosure relates a decodingdevice, comprising one or more processing circuitry configured to:

parse from a bitstream a syntax element for indicating whether thepreviously shown method is enable or not; and

adaptively enabling or disabling a decoder-side motion vector refinement(DMVR) procedure according to the syntax element.

In a possible implementation form of the method according to thefifteenth aspect as such, wherein the syntax element is obtained fromany one of a sequence parameter set (SPS) level of a bitstream, apicture parameter set (PPS) level of the bitstream, a slice header,coding tree unit (CTU) syntax, or coding unit (CU) syntax.

In a possible implementation form of the method according to anypreceding implementation of the fifteenth aspect or the fifteenth aspectas such, wherein the syntax element comprises the first flag(sps_conditional_dmvr_flag) and the second flag(pps_conditional_dmvr_flag);

if the first flag (sps_conditional_dmvr_flag) is equal to 0, thepreviously shown method is not performed for the image blocks of thesequence;if the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 0, the previouslyshown method is not performed for the image blocks of the picture of thesequence; orif the first flag (sps_conditional_dmvr_flag) is equal to 1 and thesecond flag (pps_conditional_dmvr_flag) is equal to 1, the previouslyshown method is performed for the image blocks of the picture of thesequence.

According to a sixteenth aspect, the disclosure relatescomputer-readable medium storing computer-readable instructions whichwhen executed on a processor perform the steps according to thepreviously shown methods.

According to a seventeenth aspect, the disclosure relates a method forinter-prediction of a current image block in a current picture of avideo, the method comprising:

determining, whether at least one condition is met, wherein the at leastone condition comprises: a first temporal distance (such as TD0) isequal to a second temporal distance (such as TD1), wherein the firsttemporal distance is represented in terms of difference between thepicture order count value of the current picture and the picture ordercount value of the first reference image; and the second temporaldistance is represented in terms of difference between the picture ordercount value of the second reference image and the picture order countvalue of the current picture; andperforming, when the at least one condition is met, decoder side motionvector refinement (DMVR) procedure to determine a prediction block ofthe current image block.

According to an eighteenth aspect, the disclosure relates a method forinter-prediction of a current image block in a current picture of avideo, the method comprising:

performing, when one or more conditions are met, motion vectorrefinement (DMVR) procedure to obtain a first refined motion vector anda second refined motion vector for a sub-block of the current imageblock, wherein the first refined motion vector corresponds to a firstreference picture and the second refined motion vector corresponds to asecond reference picture; anddetermining a prediction block (such as predicted sample values) of thecurrent image block, wherein the prediction block of the current imageblock comprises a prediction block of the sub-block, wherein theprediction block of the sub-block is determined based at least in partupon the first refined motion vector and the second refined motionvector;wherein the one or more conditions at least comprises:a first temporal distance (such as TD0) is equal to a second temporaldistance (such as TD1), wherein the first temporal distance (TD0) isrepresented in terms of difference between the picture order count value(POCc) of the current picture and the picture order count value (POC0)of the first reference image; and the second temporal distance (TD1) isrepresented in terms of difference between the picture order count value(POCc) of the second reference picture and the picture order count value(POC1) of the current image.

In a possible implementation form of the method according to anypreceding implementation of the fifteenth, sixteenth, seventeenth andeighteenth aspect or the fifteenth, sixteenth, seventeenth andeighteenth aspect as such, wherein the first reference picture is areference image temporally preceding the current picture and the secondreference picture is a reference image preceded temporally by thecurrent picture; or the first reference picture is a reference imagepreceded temporally by the current picture and the second referencepicture is a reference image temporally preceding the current picture;or

wherein the first reference picture is a reference image in the past andthe second reference picture is a reference image in the future; or thefirst reference picture is a reference image in the future and thesecond reference picture is a reference image in the past.

The method according to some aspect of the invention can be performed bythe apparatus according to the some aspect of the invention. Furtherfeatures and implementation forms of the method according to the someaspect of the invention result directly from the functionality of theapparatus according to the some aspect of the invention and itsdifferent implementation forms.

It is noted that a coding device may be encoding device or decodingdevice.

According to another aspect the invention relates to an apparatus fordecoding a video stream includes a processor and a memory. The memory isstoring instructions that cause the processor to perform the previouslyshown method.

According to another aspect the invention relates to an apparatus forencoding a video stream includes a processor and a memory. The memory isstoring instructions that cause the processor to perform the previouslyshown method.

According to another aspect, a computer-readable storage medium havingstored thereon instructions that when executed cause one or moreprocessors configured to code video data is proposed. The instructionscause the one or more processors to perform a previously shown method.

According to another aspect, a computer program product with a programcode for performing the previously shown method when the computerprogram runs on a computer, is provided.

Details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description, drawings, and claims.

For the purpose of clarity, any one of the foregoing embodiments may becombined with any one or more of the other foregoing embodiments tocreate a new embodiment within the scope of the present disclosure.

These and other features will be more clearly understood from thefollowing detailed description taken in conjunction with theaccompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure, reference is nowmade to the following brief description, taken in connection with theaccompanying drawings and detailed description, wherein like referencenumerals represent like parts.

FIG. 1 is a block diagram illustrating an example coding system that mayutilize conditional decoder-side motion vector refinement techniques.

FIG. 2 is a block diagram illustrating an example video encoder that mayimplement conditional decoder-side motion vector refinement techniques.

FIG. 3 is a block diagram illustrating an example of a video decoderthat may implement conditional decoder-side motion vector refinementtechniques.

FIG. 4 is a schematic diagram of a coding device.

FIG. 5 is a block diagram illustrating an example of an encodingapparatus or a decoding apparatus.

FIG. 6A is a graphical illustration of the temporal distances forconditional decoder-side motion vector refinement in video coding.

FIG. 6B is a graphical illustration of an example of a decoder-sidemotion vector refinement (DMVR) procedure.

FIG. 6C is a flowchart illustrating an example of a decoder-side motionvector refinement (DMVR) procedure in reference with FIG. 6B.

FIG. 6D is a flowchart illustrating another example of a decoder-sidemotion vector refinement (DMVR) procedure.

FIG. 7 is a flowchart illustrating an example of an encoding method.

FIG. 8 is a flowchart illustrating an example of a decoding method.

FIG. 9 is a flowchart illustrating an example of a method forinter-prediction of a current image block in a current picture of avideo.

FIG. 10 is a block diagram showing an example structure of an apparatusfor inter-prediction of a current image block in a current picture of avideo.

FIG. 11 is a block diagram showing an example structure of a contentsupply system which provides a content delivery service.

FIG. 12 is a block diagram showing a structure of an example of aterminal device.

DETAILED DESCRIPTION

It should be understood at the outset that although an illustrativeimplementation of one or more embodiments are provided below, thedisclosed systems and/or methods may be implemented using any number oftechniques, whether currently known or in existence. The disclosureshould in no way be limited to the illustrative implementations,drawings, and techniques illustrated below, including the exemplarydesigns and implementations illustrated and described herein, but may bemodified within the scope of the appended claims along with their fullscope of equivalents.

DEFINITIONS OF ACRONYMS & GLOSSARIES DMVR Decoder Side Motion VectorRefinement SAD Sum of Absolute Differences MV Motion Vector MCP MotionCompensated Prediction HEVC High Efficient Video Coding

FIG. 1 is a block diagram illustrating an example coding system 10 thatmay utilize bidirectional prediction techniques. As shown in FIG. 1, thecoding system 10 includes a source device 12 that provides encoded videodata to be decoded at a later time by a destination device 14. Inparticular, the source device 12 may provide the video data todestination device 14 via a computer-readable medium 16. Source device12 and destination device 14 may comprise any of a wide range ofdevices, including desktop computers, notebook (i.e., laptop) computers,tablet computers, set-top boxes, telephone handsets such as so-called“smart” phones, so-called “smart” pads, televisions, cameras, displaydevices, digital media players, video gaming consoles, video streamingdevice, or the like. In some cases, source device 12 and destinationdevice 14 may be equipped for wireless communication.

Destination device 14 may receive the encoded video data to be decodedvia computer-readable medium 16. Computer-readable medium 16 maycomprise any type of medium or device capable of moving the encodedvideo data from source device 12 to destination device 14. In oneexample, computer-readable medium 16 may comprise a communication mediumto enable source device 12 to transmit encoded video data directly todestination device 14 in real-time. The encoded video data may bemodulated according to a communication standard, such as a wirelesscommunication protocol, and transmitted to destination device 14. Thecommunication medium may comprise any wireless or wired communicationmedium, such as a radio frequency (RF) spectrum or one or more physicaltransmission lines. The communication medium may form part of apacket-based network, such as a local area network, a wide-area network,or a global network such as the Internet. The communication medium mayinclude routers, switches, base stations, or any other equipment thatmay be useful to facilitate communication from source device 12 todestination device 14.

In some examples, encoded data may be output from output interface 22 toa storage device. Similarly, encoded data may be accessed from thestorage device by input interface. The storage device may include any ofa variety of distributed or locally accessed data storage media such asa hard drive, Blu-ray discs, digital video disks (DVD)s, Compact DiscRead-Only Memories (CD-ROMs), flash memory, volatile or non-volatilememory, or any other suitable digital storage media for storing encodedvideo data. In a further example, the storage device may correspond to afile server or another intermediate storage device that may store theencoded video generated by source device 12. Destination device 14 mayaccess stored video data from the storage device via streaming ordownload. The file server may be any type of server capable of storingencoded video data and transmitting that encoded video data to thedestination device 14. Example file servers include a web server (e.g.,for a website), a file transfer protocol (FTP) server, network attachedstorage (NAS) devices, or a local disk drive. Destination device 14 mayaccess the encoded video data through any standard data connection,including an Internet connection. This may include a wireless channel(e.g., a Wi-Fi connection), a wired connection (e.g., digital subscriberline (DSL), cable modem, etc.), or a combination of both that issuitable for accessing encoded video data stored on a file server. Thetransmission of encoded video data from the storage device may be astreaming transmission, a download transmission, or a combinationthereof.

The techniques of this disclosure are not necessarily limited towireless applications or settings. The techniques may be applied tovideo coding in support of any of a variety of multimedia applications,such as over-the-air television broadcasts, cable televisiontransmissions, satellite television transmissions, Internet streamingvideo transmissions, such as dynamic adaptive streaming over HTTP(DASH), digital video that is encoded onto a data storage medium,decoding of digital video stored on a data storage medium, or otherapplications. In some examples, coding system 10 may be configured tosupport one-way or two-way video transmission to support applicationssuch as video streaming, video playback, video broadcasting, and/orvideo telephony.

In the example of FIG. 1, source device 12 includes video source 18,video encoder 20, and output interface 22. Destination device 14includes input interface 28, video decoder 30, and display device 32. Inaccordance with this disclosure, video encoder 20 of source device 12and/or the video decoder 30 of the destination device 14 may beconfigured to apply the techniques for bidirectional prediction. Inother examples, a source device and a destination device may includeother components or arrangements. For example, source device 12 mayreceive video data from an external video source, such as an externalcamera. Likewise, destination device 14 may interface with an externaldisplay device, rather than including an integrated display device.

The illustrated coding system 10 of FIG. 1 is merely one example.Techniques for bidirectional prediction may be performed by any digitalvideo encoding and/or decoding device. Although the techniques of thisdisclosure generally are performed by a video coding device, thetechniques may also be performed by a video encoder/decoder, typicallyreferred to as a “CODEC.” Moreover, the techniques of this disclosuremay also be performed by a video preprocessor. The video encoder and/orthe decoder may be a graphics processing unit (GPU) or a similar device.

Source device 12 and destination device 14 are merely examples of suchcoding devices in which source device 12 generates coded video data fortransmission to destination device 14. In some examples, source device12 and destination device 14 may operate in a substantially symmetricalmanner such that each of the source and destination devices 12, 14includes video encoding and decoding components. Hence, coding system 10may support one-way or two-way video transmission between video devices12, 14, e.g., for video streaming, video playback, video broadcasting,or video telephony.

Video source 18 of source device 12 may include a video capture device,such as a video camera, a video archive containing previously capturedvideo, and/or a video feed interface to receive video from a videocontent provider. As a further alternative, video source 18 may generatecomputer graphics-based data as the source video, or a combination oflive video, archived video, and computer-generated video.

In some cases, when video source 18 is a video camera, source device 12and destination device 14 may form so-called camera phones or videophones. As mentioned above, however, the techniques described in thisdisclosure may be applicable to video coding in general, and may beapplied to wireless and/or wired applications. In each case, thecaptured, pre-captured, or computer-generated video may be encoded byvideo encoder 20. The encoded video information may then be output byoutput interface 22 onto a computer-readable medium 16.

Computer-readable medium 16 may include transient media, such as awireless broadcast or wired network transmission, or storage media (thatis, non-transitory storage media), such as a hard disk, flash drive,compact disc, digital video disc, Blu-ray disc, or othercomputer-readable media. In some examples, a network server (not shown)may receive encoded video data from source device 12 and provide theencoded video data to destination device 14, e.g., via networktransmission. Similarly, a computing device of a medium productionfacility, such as a disc stamping facility, may receive encoded videodata from source device 12 and produce a disc containing the encodedvideo data. Therefore, computer-readable medium 16 may be understood toinclude one or more computer-readable media of various forms, in variousexamples.

Input interface 28 of destination device 14 receives information fromcomputer-readable medium 16. The information of computer-readable medium16 may include syntax information defined by video encoder 20, which isalso used by video decoder 30, that includes syntax elements thatdescribe characteristics and/or processing of blocks and other codedunits, e.g., group of pictures (GOPs). Display device 32 displays thedecoded video data to a user, and may comprise any of a variety ofdisplay devices such as a cathode ray tube (CRT), a liquid crystaldisplay (LCD), a plasma display, an organic light emitting diode (OLED)display, or another type of display device.

Video encoder 20 and video decoder 30 may operate according to a videocoding standard, such as the High Efficiency Video Coding (HEVC)standard presently under development, and may conform to the HEVC TestModel (HM). Alternatively, video encoder 20 and video decoder 30 mayoperate according to other proprietary or industry standards, such asthe International Telecommunications Union TelecommunicationStandardization Sector (ITU-T) H.264 standard, alternatively referred toas Motion Picture Expert Group (MPEG)-4, Part 10, Advanced Video Coding(AVC), H.265/HEVC, or extensions of such standards. The techniques ofthis disclosure, however, are not limited to any particular codingstandard. Other examples of video coding standards include MPEG-2 andITU-T H.263. Although not shown in FIG. 1, in some aspects, videoencoder 20 and video decoder 30 may each be integrated with an audioencoder and decoder, and may include appropriatemultiplexer-demultiplexer (MUX-DEMUX) units, or other hardware andsoftware, to handle encoding of both audio and video in a common datastream or separate data streams. If applicable, MUX-DEMUX units mayconform to the ITU H.223 multiplexer protocol, or other protocols suchas the user datagram protocol (UDP).

Video encoder 20 and video decoder 30 each may be implemented as any ofa variety of suitable encoder circuitry, such as one or moremicroprocessors, digital signal processors (DSPs), application specificintegrated circuits (ASICs), field programmable gate arrays (FPGAs),discrete logic, software, hardware, firmware or any combinationsthereof. When the techniques are implemented partially in software, adevice may store instructions for the software in a suitable,non-transitory computer-readable medium and execute the instructions inhardware using one or more processors to perform the techniques of thisdisclosure. Each of video encoder 20 and video decoder 30 may beincluded in one or more encoders or decoders, either of which may beintegrated as part of a combined encoder/decoder (CODEC) in a respectivedevice. A device including video encoder 20 and/or video decoder 30 maycomprise an integrated circuit, a microprocessor, and/or a wirelesscommunication device, such as a cellular telephone.

FIG. 2 is a block diagram illustrating an example of video encoder 20that may implement bidirectional prediction techniques. Video encoder 20may perform intra- and inter-coding of video blocks within video slices.Intra-coding relies on spatial prediction to reduce or remove spatialredundancy in video within a given video frame or picture. Inter-codingrelies on temporal prediction to reduce or remove temporal redundancy invideo within adjacent frames or pictures of a video sequence. Intra-mode(I mode) may refer to any of several spatial based coding modes.Inter-modes, such as uni-directional prediction (P mode) orbi-prediction (B mode), may refer to any of several temporal-basedcoding modes.

As shown in FIG. 2, video encoder 20 receives a current video blockwithin a video frame to be encoded. In the example of FIG. 2, videoencoder 20 includes mode select unit 40, reference frame memory 64,summer 50, transform processing unit 52, quantization unit 54, andentropy coding unit 56. Mode select unit 40, in turn, includes motioncompensation unit 44, motion estimation unit 42, intra-prediction unit46, and partition unit 48. For video block reconstruction, video encoder20 also includes inverse quantization unit 58, inverse transform unit60, and summer 62. A deblocking filter (not shown in FIG. 2) may also beincluded to filter block boundaries to remove blockiness artifacts fromreconstructed video. If desired, the deblocking filter would typicallyfilter the output of summer 62. Additional filters (in loop or postloop) may also be used in addition to the deblocking filter. Suchfilters are not shown for brevity, but if desired, may filter the outputof summer 50 (as an in-loop filter).

During the encoding process, video encoder 20 receives a video frame orslice to be coded. The frame or slice may be divided into multiple videoblocks. Motion estimation unit 42 and motion compensation unit 44perform inter-predictive coding of the received video block relative toone or more blocks in one or more reference frames to provide temporalprediction. Intra-prediction unit 46 may alternatively performintra-predictive coding of the received video block relative to one ormore neighboring blocks in the same frame or slice as the block to becoded to provide spatial prediction. Video encoder 20 may performmultiple coding passes, e.g., to select an appropriate coding mode foreach block of video data.

Moreover, partition unit 48 may partition blocks of video data intosub-blocks, based on evaluation of previous partitioning schemes inprevious coding passes. For example, partition unit 48 may initiallypartition a frame or slice into largest coding units (LCUs), andpartition each of the LCUs into sub-coding units (sub-CUs) based onrate-distortion analysis (e.g., rate-distortion optimization). Modeselect unit 40 may further produce a quadtree data structure indicativeof partitioning of a LCU into sub-CUs. Leaf-node CUs of the quadtree mayinclude one or more prediction units (PUs) and one or more transformunits (TUs).

The present disclosure uses the term “block” to refer to any of a CU,PU, or TU, in the context of HEVC, or similar data structures in thecontext of other standards (e.g., macroblocks and sub-blocks thereof inH.264/AVC). A CU includes a coding node, PUs, and TUs associated withthe coding node. A size of the CU corresponds to a size of the codingnode and is square in shape. The size of the CU may range from 8×8pixels up to the size of the treeblock with a maximum of 64×64 pixels orgreater. Each CU may contain one or more PUs and one or more TUs. Syntaxdata associated with a CU may describe, for example, partitioning of theCU into one or more PUs. Partitioning modes may differ between whetherthe CU is skip or direct mode encoded, intra-prediction mode encoded, orinter-prediction mode encoded. PUs may be partitioned to be non-squarein shape. Syntax data associated with a CU may also describe, forexample, partitioning of the CU into one or more TUs according to aquadtree. In an embodiment, a CU, PU, or TU can be square or non-square(e.g., rectangular) in shape.

Mode select unit 40 may select one of the coding modes, intra or inter,e.g., based on error results, and provides the resulting intra- orinter-coded block to summer 50 to generate residual block data and tosummer 62 to reconstruct the encoded block for use as a reference frame.Mode select unit 40 also provides syntax elements, such as motionvectors, intra-mode indicators, partition information, and other suchsyntax information, to entropy coding unit 56.

Motion estimation unit 42 and motion compensation unit 44 may be highlyintegrated, but are illustrated separately for conceptual purposes.Motion estimation, performed by motion estimation unit 42, is theprocess of generating motion vectors, which estimate motion for videoblocks. A motion vector, for example, may indicate the displacement of aPU of a video block within a current video frame or picture relative toa predictive block within a reference frame (or other coded unit)relative to the current block being coded within the current frame (orother coded unit). A predictive block is a block that is found toclosely match the block to be coded, in terms of pixel difference, whichmay be determined by sum of absolute difference (SAD), sum of squaredifference (SSD), or other difference metrics. In some examples, videoencoder 20 may calculate values for sub-integer pixel positions ofreference pictures stored in reference frame memory 64. For example,video encoder 20 may interpolate values of one-quarter pixel positions,one-eighth pixel positions, or other fractional pixel positions of thereference picture. Therefore, motion estimation unit 42 may perform amotion search relative to the full pixel positions and fractional pixelpositions and output a motion vector with fractional pixel precision.

Motion estimation unit 42 calculates a motion vector for a PU of a videoblock in an inter-coded slice by comparing the position of the PU to theposition of a predictive block of a reference picture. The referencepicture may be selected from a first reference picture list (List 0) ora second reference picture list (List 1), each of which identify one ormore reference pictures stored in reference frame memory 64. Motionestimation unit 42 sends the calculated motion vector to entropyencoding unit 56 and motion compensation unit 44.

Motion compensation, performed by motion compensation unit 44, mayinvolve fetching or generating the predictive block based on the motionvector determined by motion estimation unit 42. Again, motion estimationunit 42 and motion compensation unit 44 may be functionally integrated,in some examples. Upon receiving the motion vector for the PU of thecurrent video block, motion compensation unit 44 may locate thepredictive block to which the motion vector points in one of thereference picture lists. Summer 50 forms a residual video block bysubtracting pixel values of the predictive block from the pixel valuesof the current video block being coded, forming pixel difference values,as discussed below. In general, motion estimation unit 42 performsmotion estimation relative to luma components, and motion compensationunit 44 uses motion vectors calculated based on the luma components forboth chroma components and luma components. Mode select unit 40 may alsogenerate syntax elements associated with the video blocks and the videoslice for use by video decoder 30 in decoding the video blocks of thevideo slice.

Intra-prediction unit 46 may intra-predict a current block, as analternative to the inter-prediction performed by motion estimation unit42 and motion compensation unit 44, as described above. In particular,intra-prediction unit 46 may determine an intra-prediction mode to useto encode a current block. In some examples, intra-prediction unit 46may encode a current block using various intra-prediction modes, e.g.,during separate encoding passes, and intra-prediction unit 46 (or modeselect unit 40, in some examples) may select an appropriateintra-prediction mode to use from the tested modes.

For example, intra-prediction unit 46 may calculate rate-distortionvalues using a rate-distortion analysis for the various testedintra-prediction modes, and select the intra-prediction mode having thebest rate-distortion characteristics among the tested modes.Rate-distortion analysis generally determines an amount of distortion(or error) between an encoded block and an original, unencoded blockthat was encoded to produce the encoded block, as well as a bitrate(that is, a number of bits) used to produce the encoded block.Intra-prediction unit 46 may calculate ratios from the distortions andrates for the various encoded blocks to determine which intra-predictionmode exhibits the best rate-distortion value for the block.

In addition, intra-prediction unit 46 may be configured to code depthblocks of a depth map using a depth modeling mode (DMM). Mode selectunit 40 may determine whether an available DMM mode produces bettercoding results than an intra-prediction mode and the other DMM modes,e.g., using rate-distortion optimization (RDO). Data for a texture imagecorresponding to a depth map may be stored in reference frame memory 64.Motion estimation unit 42 and motion compensation unit 44 may also beconfigured to inter-predict depth blocks of a depth map.

After selecting an intra-prediction mode for a block (e.g., aconventional intra-prediction mode or one of the DMM modes),intra-prediction unit 46 may provide information indicative of theselected intra-prediction mode for the block to entropy coding unit 56.Entropy coding unit 56 may encode the information indicating theselected intra-prediction mode. Video encoder 20 may include in thetransmitted bitstream configuration data, which may include a pluralityof intra-prediction mode index tables and a plurality of modifiedintra-prediction mode index tables (also referred to as codeword mappingtables), definitions of encoding contexts for various blocks, andindications of a most probable intra-prediction mode, anintra-prediction mode index table, and a modified intra-prediction modeindex table to use for each of the contexts.

Video encoder 20 forms a residual video block by subtracting theprediction data from mode select unit 40 from the original video blockbeing coded. Summer 50 represents the component or components thatperform this subtraction operation.

Transform processing unit 52 applies a transform, such as a discretecosine transform (DCT) or a conceptually similar transform, to theresidual block, producing a video block comprising residual transformcoefficient values. Transform processing unit 52 may perform othertransforms which are conceptually similar to DCT. Wavelet transforms,integer transforms, sub-band transforms or other types of transformscould also be used.

Transform processing unit 52 applies the transform to the residualblock, producing a block of residual transform coefficients. Thetransform may convert the residual information from a pixel value domainto a transform domain, such as a frequency domain. Transform processingunit 52 may send the resulting transform coefficients to quantizationunit 54. Quantization unit 54 quantizes the transform coefficients tofurther reduce bit rate. The quantization process may reduce the bitdepth associated with some or all of the coefficients. The degree ofquantization may be modified by adjusting a quantization parameter. Insome examples, quantization unit 54 may then perform a scan of thematrix including the quantized transform coefficients. Alternatively,entropy encoding unit 56 may perform the scan.

Following quantization, entropy coding unit 56 entropy codes thequantized transform coefficients. For example, entropy coding unit 56may perform context adaptive variable length coding (CAVLC), contextadaptive binary arithmetic coding (CABAC), syntax-based context-adaptivebinary arithmetic coding (SBAC), probability interval partitioningentropy (PIPE) coding or another entropy coding technique. In the caseof context-based entropy coding, context may be based on neighboringblocks. Following the entropy coding by entropy coding unit 56, theencoded bitstream may be transmitted to another device (e.g., videodecoder 30) or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform unit 60 apply inversequantization and inverse transformation, respectively, to reconstructthe residual block in the pixel domain, e.g., for later use as areference block. Motion compensation unit 44 may calculate a referenceblock by adding the residual block to a predictive block of one of theframes of reference frame memory 64. Motion compensation unit 44 mayalso apply one or more interpolation filters to the reconstructedresidual block to calculate sub-integer pixel values for use in motionestimation. Summer 62 adds the reconstructed residual block to themotion compensated prediction block produced by motion compensation unit44 to produce a reconstructed video block for storage in reference framememory 64. The reconstructed video block may be used by motionestimation unit 42 and motion compensation unit 44 as a reference blockto inter-code a block in a subsequent video frame.

Other structural variations of the video encoder 20 can be used toencode the video stream. For example, a non-transform based encoder 20can quantize the residual signal directly without the transformprocessing unit 52 for certain blocks or frames. In anotherimplementation, an encoder 20 can have the quantization unit 54 and theinverse quantization unit 58 combined into a single unit.

FIG. 3 is a block diagram illustrating an example of video decoder 30that may implement bidirectional prediction techniques. In the exampleof FIG. 3, video decoder 30 includes an entropy decoding unit 70, motioncompensation unit 72, intra-prediction unit 74, inverse quantizationunit 76, inverse transformation unit 78, reference frame memory 82, andsummer 80. Video decoder 30 may, in some examples, perform a decodingpass generally reciprocal to the encoding pass described with respect tovideo encoder 20 (FIG. 2). Motion compensation unit 72 may generateprediction data based on motion vectors received from entropy decodingunit 70, while intra-prediction unit 74 may generate prediction databased on intra-prediction mode indicators received from entropy decodingunit 70.

During the decoding process, video decoder 30 receives an encoded videobitstream that represents video blocks of an encoded video slice andassociated syntax elements from video encoder 20. Entropy decoding unit70 of video decoder 30 entropy decodes the bitstream to generatequantized coefficients, motion vectors or intra-prediction modeindicators, and other syntax elements. Entropy decoding unit 70 forwardsthe motion vectors and other syntax elements to motion compensation unit72. Video decoder 30 may receive the syntax elements at the video slicelevel and/or the video block level.

When the video slice is coded as an intra-coded (I) slice, intraprediction unit 74 may generate prediction data for a video block of thecurrent video slice based on a signaled intra prediction mode and datafrom previously decoded blocks of the current frame or picture. When thevideo frame is coded as an inter-coded (i.e., B, P, or GPB) slice,motion compensation unit 72 produces predictive blocks for a video blockof the current video slice based on the motion vectors and other syntaxelements received from entropy decoding unit 70. The predictive blocksmay be produced from one of the reference pictures within one of thereference picture lists. Video decoder 30 may construct the referenceframe lists, List 0 and List 1, using default construction techniquesbased on reference pictures stored in reference frame memory 82.

Motion compensation unit 72 determines prediction information for avideo block of the current video slice by parsing the motion vectors andother syntax elements, and uses the prediction information to producethe predictive blocks for the current video block being decoded. Forexample, motion compensation unit 72 uses some of the received syntaxelements to determine a prediction mode (e.g., intra- orinter-prediction) used to code the video blocks of the video slice, aninter-prediction slice type (e.g., B slice, P slice, or GPB slice),construction information for one or more of the reference picture listsfor the slice, motion vectors for each inter-encoded video block of theslice, inter-prediction status for each inter-coded video block of theslice, and other information to decode the video blocks in the currentvideo slice.

Motion compensation unit 72 may also perform interpolation based oninterpolation filters. Motion compensation unit 72 may use interpolationfilters as used by video encoder 20 during encoding of the video blocksto calculate interpolated values for sub-integer pixels of referenceblocks. In this case, motion compensation unit 72 may determine theinterpolation filters used by video encoder 20 from the received syntaxelements and use the interpolation filters to produce predictive blocks.

Data for a texture image corresponding to a depth map may be stored inreference frame memory 82. Motion compensation unit 72 may also beconfigured to inter-predict depth blocks of a depth map.

As will be appreciated by those in the art, the coding system 10 of FIG.1 is suitable for implementing various video coding or compressiontechniques. Some video compression techniques, such as inter prediction,intra prediction, and loop filters, have demonstrated to be effective.Therefore, the video compression techniques have been adopted intovarious video coding standards, such as H.264/AVC and H.265/IEVC.

Various coding tools such as adaptive motion vector prediction (AMVP)and merge mode (MERGE) are used to predict motion vectors (MVs) andenhance inter prediction efficiency and, therefore, the overall videocompression efficiency.

The MVs noted above are utilized in bi-prediction. In a bi-predictionoperation, two prediction blocks are formed. One prediction block isformed using a MV of list0 (referred to herein as MV0). Anotherprediction block is formed using a MV of list1 (referred to herein asMV1). The two prediction blocks are then combined (e.g., averaged) inorder to form a single prediction signal (e.g., a prediction block or apredictor block).

Other variations of the video decoder 30 can be used to decode thecompressed bitstream. For example, the decoder 30 can produce the outputvideo stream without the loop filtering unit. For example, anon-transform based decoder 30 can inverse-quantize the residual signaldirectly without the inverse-transform processing unit 78 for certainblocks or frames. In another implementation, the video decoder 30 canhave the inverse-quantization unit 76 and the inverse-transformprocessing unit 78 combined into a single unit.

FIG. 5 is a schematic diagram of a coding device 500 according to anembodiment of the disclosure. The coding device 500 is suitable forimplementing the disclosed embodiments as described herein. In anembodiment, the coding device 500 may be a decoder such as video decoder30 of FIG. 1 or an encoder such as video encoder 20 of FIG. 1. In anembodiment, the coding device 500 may be one or more components of thevideo decoder 30 of FIG. 1 or the video encoder 20 of FIG. 1 asdescribed above.

The coding device 500 comprises ingress ports 510 and receiver units(Rx) 520 for receiving data; a processor, logic unit, or centralprocessing unit (CPU) 530 to process the data; transmitter units (Tx)540 and egress ports 550 for transmitting the data; and a memory 560 forstoring the data. The coding device 500 may also compriseoptical-to-electrical (OE) components and electrical-to-optical (EO)components coupled to the ingress ports 510, the receiver units 520, thetransmitter units 540, and the egress ports 550 for egress or ingress ofoptical or electrical signals.

The processor 530 is implemented by hardware and software. The processor530 may be implemented as one or more CPU chips, cores (e.g., as amulti-core processor), FPGAs, ASICs, and DSPs. The processor 530 is incommunication with the ingress ports 510, receiver units 520,transmitter units 540, egress ports 550, and memory 560. The processor530 comprises a coding module 570. The coding module 570 implements thedisclosed embodiments described above. For instance, the coding module570 implements, processes, prepares, or provides the various codingoperations. The inclusion of the coding module 570 therefore provides asubstantial improvement to the functionality of the coding device 500and effects a transformation of the coding device 500 to a differentstate. Alternatively, the coding module 570 is implemented asinstructions stored in the memory 560 and executed by the processor 530.

The memory 560 comprises one or more disks, tape drives, and solid-statedrives and may be used as an over-flow data storage device, to storeprograms when such programs are selected for execution, and to storeinstructions and data that are read during program execution. The memory560 may be volatile and/or non-volatile and may be read-only memory(ROM), random access memory (RAM), ternary content-addressable memory(TCAM), and/or static random-access memory (SRAM).

FIG. 4 is a simplified block diagram of an apparatus 1000 that may beused as either or both of the source device 12 and the destinationdevice 14 from FIG. 1A according to an exemplary embodiment. Theapparatus 1000 can implement techniques of this present application. Theapparatus 1000 can be in the form of a computing system includingmultiple computing devices, or in the form of a single computing device,for example, a mobile phone, a tablet computer, a laptop computer, anotebook computer, a desktop computer, and the like.

A processor 1002 in the apparatus 1000 can be a central processing unit.Alternatively, the processor 1002 can be any other type of device, ormultiple devices, capable of manipulating or processing informationnow-existing or hereafter developed. Although the disclosedimplementations can be practiced with a single processor as shown, e.g.,the processor 1002, advantages in speed and efficiency can be achievedusing more than one processor.

A memory 1004 in the apparatus 1000 can be a read only memory (ROM)device or a random access memory (RAM) device in an implementation. Anyother suitable type of storage device can be used as the memory 1004.The memory 1004 can include code and data 1006 that is accessed by theprocessor 1002 using a bus 1012. The memory 1004 can further include anoperating system 1008 and application programs 1010, the applicationprograms 1010 including at least one program that permits the processor1002 to perform the methods described here. For example, the applicationprograms 1010 can include applications 1 through N, which furtherinclude a video coding application that performs the methods describedhere. The apparatus 1000 can also include additional memory in the formof a secondary storage 1014, which can, for example, be a memory cardused with a mobile computing device. Because the video communicationsessions may contain a significant amount of information, they can bestored in whole or in part in the secondary storage 1014 and loaded intothe memory 1004 as needed for processing.

The apparatus 1000 can also include one or more output devices, such asa display 1018. The display 1018 may be, in one example, a touchsensitive display that combines a display with a touch sensitive elementthat is operable to sense touch inputs. The display 1018 can be coupledto the processor 1002 via the bus 1012. Other output devices that permita user to program or otherwise use the apparatus 1000 can be provided inaddition to or as an alternative to the display 1018. When the outputdevice is or includes a display, the display can be implemented invarious ways, including by a liquid crystal display (LCD), a cathode-raytube (CRT) display, a plasma display or light emitting diode (LED)display, such as an organic LED (OLED) display.

The apparatus 1000 can also include or be in communication with animage-sensing device 1020, for example a camera, or any otherimage-sensing device 1020 now existing or hereafter developed that cansense an image such as the image of a user operating the apparatus 1000.The image-sensing device 1020 can be positioned such that it is directedtoward the user operating the apparatus 1000. In an example, theposition and optical axis of the image-sensing device 1020 can beconfigured such that the field of vision includes an area that isdirectly adjacent to the display 1018 and from which the display 1018 isvisible.

The apparatus 1000 can also include or be in communication with asound-sensing device 1022, for example a microphone, or any othersound-sensing device now existing or hereafter developed that can sensesounds near the apparatus 1000. The sound-sensing device 1022 can bepositioned such that it is directed toward the user operating theapparatus 1000 and can be configured to receive sounds, for example,speech or other utterances, made by the user while the user operates theapparatus 1000.

Although FIG. 4 depicts the processor 1002 and the memory 1004 of theapparatus 1000 as being integrated into a single unit, otherconfigurations can be utilized. The operations of the processor 1002 canbe distributed across multiple machines (each machine having one or moreof processors) that can be coupled directly or across a local area orother network. The memory 1004 can be distributed across multiplemachines such as a network-based memory or memory in multiple machinesperforming the operations of the apparatus 1000. Although depicted hereas a single bus, the bus 1012 of the apparatus 1000 can be composed ofmultiple buses. Further, the secondary storage 1014 can be directlycoupled to the other components of the apparatus 1000 or can be accessedvia a network and can comprise a single integrated unit such as a memorycard or multiple units such as multiple memory cards. The apparatus 1000can thus be implemented in a wide variety of configurations.

In video compression, inter prediction is a process of usingreconstructed samples of previously decoded reference pictures byspecifying motion vectors relative to a current block. These motionvectors can be coded as a prediction residual by using spatial ortemporal motion vector predictors. The motion vectors can be atsub-pixel accuracy. In order to derive the sub-pixel precision pixelvalues in the reference frames from the reconstructed integer positionvalues, an interpolation filter is applied. Bi-prediction refers to aprocess where the prediction for the current block is derived as aweighted combination of two prediction blocks derived using two motionvectors from two reference picture areas. In this case, in addition tothe motion vectors, the reference indices for the reference picturesfrom which the two prediction blocks are derived also need to be coded.The motion vectors for the current block can also be derived through amerge process where a spatial neighbour's motion vectors and referenceindices are inherited without coding any motion vector residuals. Inaddition to spatial neighbours, motion vectors of previously codedreference frames are also stored and used as temporal merge options withappropriate scaling of the motion vectors to take into account thedistance to the reference frames relative to the distance to thereference frames for the current block.

Several methods have been proposed for performing a decoder-side motionvector refinement or derivation so that the motion vector residualcoding bits can be further reduced.

In a class of methods, called bilateral matching (BM) methods, themotion information of the current CU is derived by finding the closestmatch between two blocks along the motion trajectory of the current CUin two different reference pictures. This is shown in FIG. 6A. Under theassumption of continuous motion trajectory, the motion vectors MV0 andMV1 pointing to the two reference blocks shall be proportional to thetemporal distances, i.e., TD0 and TD1, between the current picture andthe two reference pictures.

In the bilateral matching merge mode, bi-prediction is always appliedsince the motion information of a CU is derived based on the closestmatch between two blocks along the motion trajectory of the current CUin two different reference pictures.

Explicit merge mode to indicate template matching merge or bilateralmatching merge can be signaled to differentiate these modes from adefault merge mode that does not require any decoder-side motion vectorderivation.

In some examples, the temporal distances are ignored and bilateralmatching is performed using motion vectors that have the equal magnitudeand opposite signs in the past and future reference frames.

In some examples, no merge index is signaled while in other examples, tosimplify the decoder complexity of performing multiple motioncompensations, an explicit merge index is signaled.

FIG. 6B is a graphical illustration of an example of a DMVR method 400.In an example, the DMVR method 400 begins with a current block 402 in acurrent picture 404. In an example, the current block 402 may be squareor non-square in shape. The current picture 404 may also be referred toas a current region, image, tile, and so on. As shown in FIG. 6B, MV0points to a first reference block (also refer to a first initialreference block) 406 in a first reference picture 408 and MV1 points toa second reference block (also refer to a second initial referenceblock) 410 in a second reference picture 412. In an example, the firstreference block 406 is ahead of the current block 402 in time, sequence,decoding order, or some other parameter. In an example, the secondreference block 410 is ahead of the current block 402 in time, sequence,decoding order, or some other parameter. The first reference block 406and the second reference block 410 may be referred to herein as initialreference blocks.

The first reference block 406 and the second reference block 410 arecombined to form the bi-lateral template block 414. In an example, thefirst reference block 406 and the second reference block 410 areaveraged together to generate the bi-lateral template block 414. In anexample, the bi-lateral template block 414 is generated as the weightedcombination of the first reference block 406 and the second referenceblock 410.

Once the bi-lateral template block 414 has been generated, a templatematching operation is performed. The template matching operationinvolves calculating a first cost between the bi-lateral template block414 and each candidate reference block in the sample region around thefirst reference block 406 and a second cost between the bi-lateraltemplate block 414 and each candidate reference block in the sampleregion around the second reference block 410. In an example, thepotential reference block that yields the corresponding lowest cost(e.g., minimum template cost) determines which reference block in eachsample region will serve as a refined reference block (a.k.a., a revisedreference block). In an example, the first and second costs aredetermined using a SAD. Other cost measures may be utilized in practicalapplications.

In the example of FIG. 6B, the first refined reference block 416 in thefirst reference picture 408, which is pointed to by MV0′, resulted inthe lowest first cost, and the second refined reference block 418 in thesecond reference picture 412, which is pointed to by MV1′, offered thelowest second cost. In an example, the first refined reference block 416and the second refined reference block 418 replace the first referenceblock 406 and the second reference block 410, respectively.

Thereafter, a prediction block 420 is generated using the first refinedreference block 416 and the second refined reference block 418. Theprediction block 420 may be referred to as a predictor block or thefinal bi-prediction results. Once generated, the prediction block 420may be used to generate an image for display on the display of anelectronic device (e.g., smart phone, tablet device, laptop computer,etc.)

The DMVR method 400 may be applied for the merge mode of bi-predictionwith one MV from a reference picture in the past and another from areference picture in the future without having to transmit additionalsyntax elements. The DMVR method 400 is not applied when localillumination compensation (LIC), affine motion, frame-rate up-conversion(FRUC), or CU merge candidate is enabled for a CU in the JointExploration Model (JEM) reference software.

In the JEM reference software, nine MV candidates (which will point tonine candidate reference blocks) are searched for each reference blockin a reference picture (e.g., for each list, e.g. list 0 or list 1). Thenine MV candidates include an original MV (namely an initial MV, e.g.initial MV0 or initial MV1) pointing to the reference block (namely aninitial reference block, e.g., reference block 406 or reference block410) and the eight MVs pointing to reference blocks around the referenceblock with one luma sample offset relative to the original MV in eithera horizontal direction, a vertical direction, or both. However, using MVcandidates having a one luma sample offset relative to the original MVmay not provide the best MV candidate.

In an example method, bilateral template matching based decoder-sidemotion vector refinement (DMVR) method is provided, a bilaterallyaveraged template is first created using the reference blocks in L0 andL1 reference obtained from explicitly signalled merge candidate indexand bilateral matching is performed against this template. This isillustrated in FIGS. 6B and 6C. The template is updated if there is anymovement between an initial reference block (406, 410) referred by theinitial motion vector and a reference block (416, 418) referred by thelatest best motion vector. Also, in some examples, the refinement isperformed in one reference and the motion vector in the other referenceis obtained through mirroring of this refined motion vector. Therefinement alternates between the two references until either the centerposition has the least matching error or the maximum number ofiterations is reached.

FIG. 6C is a flowchart illustrating an example of a coding method 600.In an example, the coding method 600 is implemented in a decoder such asthe video decoder 30 in FIG. 1. The coding method 600 may be implementedwhen, for example, a bitstream received from an encoder, such as thevideo encoder 20 of FIG. 1, is to be decoded in order to generate animage to be displayed on the display of an electronic device. The codingmethod 600 may also be implemented in an encoder such as the videoencoder 20 in FIG. 1. The coding method 600 will be described withreference to the elements identified in FIG. 6B.

In block 602, a first reference block (e.g., reference block 406) in afirst reference picture (e.g., reference picture 408) is determinedbased on a first motion vector (e.g., MV0) corresponding to a currentblock (e.g., current block 402) in a current picture (e.g., currentpicture 404).

In block 604, a second reference block (e.g., reference block 410) in asecond reference picture (e.g., reference picture 412) is determinedbased on a second motion vector (e.g., MV1) corresponding to the currentblock (e.g., current block 402) in the current picture (e.g., currentpicture 604).

In block 606, a bilateral reference block (e.g., bi-lateral referenceblock 414) is generated based on the first reference block and thesecond reference block. In an example, the bilateral reference block isobtained using a weighted average of the first reference block and thesecond reference block.

In block 608, a cost comparison between the bilateral reference blockand each of a plurality of first reference block candidates in the firstreference picture is performed. The first reference block candidates maybe, for example, the various reference blocks surrounding the firstreference block 406 in the first reference picture 408. The costcomparison is used to determine a first refined motion vector (e.g.,MV0′). In an example, the first reference block candidates aredetermined based on a step size that was selected from a plurality ofavailable step sizes (e.g., ⅛, ¼, ½, 1, and so on).

In block 610, a cost comparison between the bilateral reference blockand each of a plurality of second reference block candidates in thesecond reference picture is performed. The second reference blockcandidates may be, for example, the various reference blocks surroundingthe second reference block 410 in the second reference picture 412. Thecost comparison is used to determine a second refined motion vector(e.g., MV1′). In an example, the second reference block candidates aredetermined based on a step size that was selected from the plurality ofavailable step sizes (e.g., ⅛, ¼, ½, 1, and so on).

In block 612, a first refined reference block (e.g., refined referenceblock 416) in the first reference picture is selected based on the firstrefined motion vector and a second refined reference block (e.g.,refined reference block 418) in the second reference picture is selectedbased on the second refined motion vector.

In block 614, a prediction block (e.g., prediction block 420) isdetermined based on the first refined reference block and the secondrefined reference block.

In block 616, an image generated using the prediction block is displayedon the display of an electronic device.

In some examples of the methods of refinement, a CU level refinement isfirst performed. Then a sub-CU (namely sub-block) level multi-candidateevaluation is performed using the CU-level refined MVs as multiplecandidates. Optionally, each sub-CU can perform its own refinement withrespect to the best matching candidate. In another example, a CU levelrefinement is not performed and each sub-CU can perform its ownrefinement.

Some cost functions use a motion vector refinement distance as a biasterm.

Given the implicit decoder-side derivation or refinement process, theencoder needs to perform these steps in exactly the same manner as thedecoder in order for the encoder-side reconstruction to match with thedecoder-side reconstruction.

Typically, only luma samples are used during the decoder side motionvector refinement or derivation process. However, in some cases,chrominance is also motion compensated using the final refinement motionvectors (suitably scaled to account for any chroma downsampling) as usedfor luma motion compensation.

One other technique for motion vector refinement on the decoder-side iscall bi-directional optical flow (BIO) technique. In this method, motioncompensated interpolation is performed for a given coding unit using anormative motion compensation method that uses the samples from the tworeference frames indicated by the reference indices and motion vectorsassociated with the coding unit. In addition, horizontal and verticalgradients at the sub-pixel precision positions are evaluated from thereference samples used for motion compensation or using the motioncompensated samples themselves. A coding unit is partitioned intouniformly sized sub-blocks where the sub-block sizes can be 1×1 pixel,2×2 pixels, 4×4 pixels, etc. An optical flow based equation that relatesvarious values associated the reference frames to generate an estimateof the sample values pred_(BIO) of the sub-block in the current frame isprovided below in Eqn. (1). In Eqn. (1), (vx, vy) represents the flow ofthe sub-block from reference frame L0 to the current frame and then toreference frame L1. Gx0 and Gy0 represent the gradients in thehorizontal and vertical directions in L0, respectively. Gx1 and Gy1represent the gradients in the horizontal and vertical directions in L1,respectively. I0 and I1 represent the intensity values of the tworeference patches in L0 and L1, respectively. τ₁ and τ₀ denote thedistances of the current frame to the reference frames L0 and L1,respectively. τ₀=POC(current)−POC(Ref0), τ₁=POC(Ref1)−POC(current).

pred_(BIO)=1/2(I0+I1+vx/2(τ1Gx1−τ0Gx0)+vy/2*(τ1Gy1−τ0Gy0))  (1)

Using the individual flow equations in each time interval, thedifference between the predictor (i.e. the predicted sample values ofthe sub-block) using L0 samples and the predictor using L1 samples canbe written as:

Δ=(I0−I1)+vx(τ1Gx1+τ0Gx0)+vy(τ1Gy1+τ0Gy0)

By minimizing the difference Δ, estimates of vx and vy can be obtained.For example, by taking partial differentials of the square of thedifference Δ with respective to vx and vy and setting the differentialsto zero, equations with vx and vy as unknowns for the samples within thesub-block and for samples adjoining the sub-block are obtained. This setof over-constrained equations can be solved through least squares toobtain vx and vy estimates. Using the equation Eqn. (1) mentioned above,the computed vx, vy estimates and the gradients Gx0 and Gy0, acorrection term is added to the usual bi-predictive motion compensation.In some methods, τ0 and τ1 are assumed to be equal in these equations.

As described above, the bilateral matching based decoder-side motionvector derivation/refinement methods compute delta motion vectors (i.e.changes to the initial motion vectors MV0 and MV1) around the merge-modemotion vectors (i.e. the merge candidate which includes initial motionvectors MV0 and MV1) in the two references (used for bi-prediction). Thecomputed delta motion vectors depend on the temporal distances TD0 andTD1 from the current picture to the two references. However, since amovement of integer distance in one reference can be a movement ofnon-integer distance in the other reference due to the ratio between TD1and TD0, evaluating the refinement requires evaluation of sub-pixelprecision positions of a different phase when compared to the phase ofthe merge mode motion vectors which constitute the starting positionsfor the refinement (For example, When TD1 and TD0 are not equal, if L0pic is at a distance of 2 and L1 is at a distance of 1, the refinementmovement of 1 pixel in L0 leads to 0.5 pixel movement in L1). Thus itleads to the high computational complexity of the refinement procedure.In order to simplify the refinement procedure, the refinement proceduremay ignore TD0 and TD1 and uses delta motion vectors that are equal inmagnitude and opposite in directions in the two references so thatinteger distance movement in one reference remains as an integerdistance movement in the other reference. Multiple iterations ofrefinement are applied until either a pre-defined maximum iterationcount is reached or the center position of a given iteration turns outto be the position with the lowest cost. This method also works well incases where a hierarchical B-pictures in a dyadic configuration areemployed as used in the common test conditions adopted in VersatileVideo Coding. In such dyadic configuration, the number of pictures atthe next temporal layer are twice as many as the number of pictures inthe current temporal layer. Hence, the B-pictures have one referencefrom the past and one reference from the future that are equi-distantfrom the current picture.

However, due to occlusions in one of the references, the candidate mergemotion vector may be applicable for uni-prediction or the candidatemerge motion vectors may refer to two reference frames that have unequaldistance to the current picture. Also, in commercial encoders, strictdyadic hierarchical B-pictures may not be used as they tend to adapt thepicture-types based on temporal correlation. For example, some encodersuse 2 non-reference pictures between every two reference pictures.Certain other encoders have variable distances between picturesbelonging to the same temporal layer due to the underlying motioncharacteristics. In such cases, the use of equal and opposite deltamotion vectors based refinement fails to produce any major coding gainand can also impact the coding gain whenever an explicit flag is notused to indicate the need for refinement. However, signaling a flag forevery CU to indicate the use of refinement offsets some of the codinggains offered by the refinement.

Hence, there is a need to selectively or adaptively provide the abilityto restrict or enable the decoder-side refinement to coding units withequi-distant references, for example, whenever bilateral matching basedrefinement with equal and opposite delta motion vectors is used.

Also, whenever the BIO method assumes equal τ0 and τ1, there is a needto provide the ability to restrict or enable the BIO method to beapplied only when the temporal distances are really equal.

The present disclosure addresses the above problems by providing amethod to selectively restrict or enable a bi-predictive merge-modecoding unit coded in at least one access unit in a group of frames fromemploying decoder side motion vector refinement based on the temporaldistances of the coding unit to its two references. Such restriction canbe performed through setting a flag at the sequence parameter set levelon the encoding-side to permit decoder-side MV refinement only when thetemporal distances of the coding unit to its two references aresubstantially equal. This method is employed at both the encoding anddecoding sides when the decoder side MV refinement is enabled.

Detailed Examples of the Presented Method

The embodiment of the present disclosure provides a method forrestricting decoder-side motion vector derivation/refinement to onlyCoding Units with equal-distance references. The embodiment may be usedin cases when decoder-side MV refinement is used. For example, themethod can be used when the bilateral matching based decoder-side MVrefinement is employed with delta motion vectors that are equal inmagnitude and opposite in sign in the two references used forbi-prediction (irrespective of whether bi-lateral template is used ornot). As explained before, such methods ignore the temporal distances tothe reference frames from the current picture so that integer-distancerefinements starting from the sub-pixel precision merge motion vectorcenters can be performed using interpolation of samples at a singlesub-pixel phase in each of the references. Merge motion vector obtainedfrom a merge candidate list can be at any sub-pixel location. Forexample, sub-pixel precision may be at 1/16. The method can also be usedwhen the Bi-directional optical flow (BDOF) based decoder-side motionvector refinement does not rely the temporal distances or assumes themto be equal.

In this embodiment, the decoder-side MV refinement is employed for agiven coding unit or coding block only when the temporal distance fromthe coding block to the reference pictures used for bi-prediction aresubstantially equal in magnitude and opposite in sign or direction. FIG.7 is a flowchart illustrating an embodiment of an encoding method 700.In an embodiment, the encoding method 700 is implemented in an encodersuch as video encoder 20 in FIG. 1. In block 701, once the mergecandidate list which includes one or more merge candidate is constructedas per the normative process on the encoding or decoding side, the mergemotion vectors and their reference indices become available. In block703, for each merge candidate that indicates bi-prediction, the 2reference frames, referred to herein as L0 and L1 references,corresponding to the 2 reference indices required for bi-prediction areidentified and their temporal distances to the current-picture, namely,TD0 and TD1 are obtained. In block 705, it is determined whether thetemporal distance to the reference pictures used for bi-prediction aresubstantially equal in magnitude and opposite in sign. In block 707, inthe case that the temporal distances are substantially equal inmagnitude and opposite in sign, the encoder or decoder performs thenormative decoder-side MV refinement process for the merge candidate.The refined motion vectors are used to perform motion compensatedbi-prediction using the reference frames L0 and L1. If it is determinedin block 705 that the temporal distance to the reference pictures usedfor bi-prediction are substantially equal in magnitude and opposite insign, in block 709, the decoder-side MV refinement is skipped, and themerge motion vectors are used for bi-prediction. In block 711, if thebest merge mode candidate has a lower cost than any other modeevaluated, then the encoder will signal a merge flag as 1 for the codingunit. Additionally, the encoder can also explicitly signal the mergecandidate index for the winning merge candidate in the merge list. Insome cases, this may be implicitly derived on the decoding side.

FIG. 8 is a flowchart illustrating an embodiment of a decoding method800. In an embodiment, the decoding method 800 is implemented in adecoder such as video decoder 30 in FIG. 1. In block 801, for a codingunit, a merge flag is parsed from a received bitstream, if the mergeflag is set to true (such as 1), the merge candidate is obtained eitherby decoding the explicitly signaled merge index or by implicitlyderiving it on the decoding side. In block 803, the normative merge listconstruction process is used to arrive at the motion vector(s) andreference index (or indices) associated with the merge index. In block805, if the merge candidate indicates bi-prediction, then the 2reference frames corresponding to the 2 reference indices are identifiedand their temporal distances to the current pictures, namely TD0 andTD1, are obtained. In block 807, it is determined whether the temporaldistance to the reference pictures used for bi-prediction aresubstantially equal in magnitude and opposite in sign. In block 809, inthe case that the temporal distances are substantially equal inmagnitude and opposite in sign, the decoder performs the normativedecoder-side MV refinement process for the merge candidate. Inparticular, if adoption of decoder-side motion vector refinement isenabled at the sequence level, then the decoder-side MV refinement isperformed only when TD0 and TD1 are substantially equal in magnitude andopposite in sign. The refined motion vectors are used for performingmotion compensated bi-prediction using the reference frames L0 and L1.Otherwise, in block 811, the decoder-side MV refinement is skipped andthe motion vectors for the merge index are used to perform motioncompensated bi-prediction. In block 813, the coding block isreconstructed based on the motion compensated data, for example, a sumof the parsed residual data and the predicted data.

In another embodiment, the same checks on TD0 and TD1 also may be usedfor bi-directional optical flow (BIO) based method (where TD0corresponds to τ0 and TD1 corresponds to −τ1 described above) to enableBIO based method to a given coding unit only when TD0 and TD1 aresubstantially equal and have opposite signs.

FIG. 9 is a flowchart of a method for inter-prediction (bi-prediction)of a current image block in a current picture of a video. The methodstarts at step 901.

At step 903, it is determined whether the current picture is temporallybetween a first reference picture (such as RefPic0) and a secondreference picture (such as RefPic1) and a first temporal distance (suchas TD0) and a second temporal distance (such as TD1) are the samedistance, wherein the first temporal distance (TD0) is between thecurrent picture and the first reference picture (RefPic0), and thesecond temporal distance (TD1) is between the current picture and thesecond reference picture (RefPic1).

At step 905, motion vector refinement (DMVR) procedure is performed todetermine a prediction block of the current image block, when it isdetermined that the current picture is temporally between the firstreference picture (such as RefPic0) and the second reference picture(such as RefPic1) and the first temporal distance (TD0) and the secondtemporal distance (TD1) are the same distance. The motion vectorrefinement (DMVR) procedure can be performed as described above withrespect to blocks 602 to 610 of FIG. 6C or blocks 632 to 640 of FIG. 6D.Other ways of performing the motion vector refinement can also beutilized.

At step 907, motion compensation is performed to obtain a predictionblock of the current image block using a first initial motion vector(MV0) and a second initial motion vector (MV1), when it is determinedthat the first temporal distance (TD0) and the second temporal distance(TD1) are different distance or the current picture is not temporallybetween the first reference picture (such as RefPic0) and the secondreference picture (such as RefPic1).

FIG. 10 is a block diagram showing an example structure of an apparatusfor inter-prediction of a current image block in a current picture of avideo. The apparatus may include:

a determining unit 101 configured to determine whether the currentpicture is temporally between a first reference picture (such asRefPic0) and a second reference picture (such as RefPic1) and a firsttemporal distance (such as TD0) and a second temporal distance (such asTD1) are the same distance, wherein the first temporal distance (TD0) isbetween the current picture and the first reference picture (RefPic0),and the second temporal distance (TD1) is between the current pictureand the second reference picture (RefPic1); andan inter prediction processing unit 103 configured to perform motionvector refinement (DMVR) procedure to determine a prediction block ofthe current image block, when it is determined that the current pictureis temporally between the first reference picture (such as RefPic0) andthe second reference picture (such as RefPic1) and the first temporaldistance (TD0) and the second temporal distance (TD1) are the samedistance.

In some embodiments, instead of always applying the above-mentionedtemporal distance based checks in order to conditionally performdecoder-side motion vector refinement at the coding unit or coding blocklevel, the checks can be conditionally performed only when a specificflag is signaled at the sequence parameter set level and/or picturelevel.

In one embodiment, a flag, such as sps_conditional_dmvr_flag, issignaled at the sequence parameter set level whenever decoder-sidemotion vector refinement is enabled at the sequence parameter set level.When this flag is set to zero, decoder-side MV refinement can beperformed independent of the temporal distances to the reference framesfrom the current picture in all access units. When this flag is set toone, an additional flag, such as pps_conditional_dmvr_flag, is signaledat the picture parameter set level. When the pps_conditional_dmvr_flagis set to zero, decoder-side MV refinement can be performed independentof the temporal distances to the reference frames from the currentpicture. When the pps_conditional_dmvr_flag is set to one, decoder-sideMV refinement can be performed only when the temporal distances to thereference frames from the current picture for a given CU aresubstantially equal in magnitude and opposite in sign.

An encoder can set sps_conditional_dmvr_flag to zero when regular dyadichierarchical B-picture group of pictures (GOP) structure is used, themaximum number of reference frames in a B-picture is set to two, and thereference picture selection always selects reference frames having equaltemporal distance to the current picture and falling on opposite sidesof the current picture. An example of a dyadic GOP structure in displayorder is I0₀ B1₄ B2₃ B3₄ B4₂ B5₄ B6₃ B7₄ B8₁ B9₄ B10₃ B11₄ B12₂ B13₄B14₃ B15₄ P16₀, where the subscript indicates the temporal layer and thenumbers next to the picture type indicate the display order frame count.

An encoder can set sps_conditional_dmvr_flag to one when (a) regulardyadic hierarchical B-picture GOP structure is used, but the maximumnumber of reference frames for a B-picture is set to be greater thantwo, or (b) if the reference picture selection is likely to selectreference pictures that do not have substantially equal temporaldistance to the current picture or that do not fall on opposite sides ofthe current picture in display order, or (c) when a non-dyadichierarchical B-pictures or non-dyadic single B-picture layer GOPstructure is used. An example of a non-dyadic GOP-structure is I0₀ B1₁B2₁ P3₀ B4₁ B5₁ P6₀ which has only one layer of B-pictures. An exampleof adaptive hierarchical B-pictures is I0₀ B1₂ B2₂ B3₁ B4₂ P5₀ where thespacing between two pictures at the same temporal layer level is decidedadaptively based on content properties.

Alternatively, sps_conditional_dmvr_flag can be configured manually asan encoder parameter by a user based on the above stated conditions.

When sps_conditional_dmvr_flag is set to 1, an encoder can setpps_conditional_dmvr_flag to zero on frames that have their maximumnumber of reference frames set to 2, and whose reference frames aresubstantially equal in temporal distance from the current picture andfall on opposite sides of the current picture in display order.

When sps_conditional_dmvr_flag is set to 1, an encoder can setpps_conditional_dmvr_flag to one on frames that have their maximumnumber of reference frames set to a value greater than 2 or in caseswhere the 2 reference frames used for bi-prediction for a CU need not beat substantially equal temporal distance from the current picture orboth the reference pictures do not fall on opposite sides of the currentpicture in display order. In an example wherein the encoding/decodingorder sequence is I0₀ P1₀, P6₀, B4₁, B2₂, B3₂, B5₂, and the displayorder is I0₀ P1₀ B2₂B3₂B4₁ B5₂P6₀, the picture B2₂ can have I0₀ P1₀,P6₀, B4₁, as its reference pictures. Among these reference pictures,reference pictures I0 and B4 are equal distance and opposite direction.Hence, when I0 and B4 are used as references for B2, the temporaldistances are equal and opposite while that is not the case when P1 andB4 are used as references. When pps_conditional_dmvr_flag is set to one,a coding unit in B2 with I0 and B4 as references will use decoder-sideMV refinement, while a coding unit in B2 with P1 and B4 as referencescannot use decoder-side MV refinement (depending on the pre-determinedratio threshold, e.g. a ratio of a distance between a current pictureand a first reference (refL0) to a distance between a second reference(refL1) and the current picture).

Since motion of an object is not necessarily linear from L0 to currentpicture and from current picture to L1, it is possible that equal andopposite motion assumption can sometimes work even when the referencepictures are not at substantially equal temporal distances. In certainembodiments, the encoder can perform the encoding for a frame withpps_conditional_dmvr_flag set to zero and another encoding for the sameframe with pps_conditional_dmvr_flag set to one and select the settingthat provides a lower rate-distortion optimized cost. Therate-distortion optimized cost is computed as a sum of a distortionmeasure of the reconstructed frame with respect to the source frame andthe bits consumed multiplied by a suitable Lagrangian multiplier thatdepends on the average quantization parameter used for the frame.

In other embodiments, the rate-distortion optimized cost can beaccumulated for both the case with decoder-side MV refinement and thecase without decoder-side MV refinement for the coding units withtemporal distances not substantially equal and the flag can be set toone for a subsequent picture if without refinement yields a loweraccumulated cost than with refinement.

When substantially equal temporal distances to reference frames is notpossible for any coding unit based on the GOP structure determined atthe encoder, it is also possible to disable decoder-side motion vectorrefinement itself at the sequence parameter set (SPS) level or pictureparameter set level. The conditional flag at the SPS level, if present,is signaled only when decoder-side motion vector refinement is enabledat the SPS level. The conditional flag at the PPS level, if present, issignaled only when decoder-side motion vector refinement is enabled atthe PPS level (explicitly or implicitly). Any alternate method ofsignaling decoder-side MV refinement at SPS/PPS level, the ability tosignal refinement unconditioned on the temporal distances to thereference, and refinement conditioned on the temporal distances to thereference is anticipated by this invention. For instance, instead of twoflags, it is possible to code a syntax element that takes one of threepossible values (e.g. 0, 1, and 2) by concatenating the two flagstogether.

FIG. 6D is a flowchart illustrating another example for performing adecoder-side motion vector refinement (DMVR) procedure or process. In anembodiment, the process is implemented in a decoder such as the videodecoder 30 in FIG. 1. The process may be implemented when, for example,a bitstream received from an encoder, such as the video encoder 20 ofFIG. 1, is to be decoded in order to generate an image on the display ofan electronic device. The process is also implemented in an encoder suchas the video encoder 20 in FIG. 1. The process will be described withreference to the elements identified in FIG. 6B.

In block 632, a position of a first initial reference block (e.g.,reference block 406) in a first reference picture (e.g., referencepicture 408) is determined based on a first motion vector (e.g., MV0)corresponding to a current block (e.g., current block 402) in a currentpicture (e.g., current picture 404).

In block 634, a position of a second initial reference block (e.g.,reference block 410) in a second reference picture (e.g., referencepicture 412) is determined based on a second motion vector (e.g., MV1)corresponding to the current block (e.g., current block 402) in thecurrent picture (e.g., current picture 404).

In block 636, positions of a plurality of first reference blocks (e.g.N−1 first reference blocks) in the first reference picture isdetermined.

In block 638, positions of a plurality of second reference blocks (e.g.N−1 second reference blocks) in the second reference picture isdetermined. In blocks 636 and 638, positions of each pair of referenceblocks includes a position of a first reference block and a position ofa second reference block, and for each pair of reference blocks, thefirst position offset (delta0x, delta0y) and the second position offset(delta1x, delta1y) are mirrored (i.e. equal in magnitude and opposite insign), and the first position offset (delta0x, delta0y) represents anoffset of the position of the first reference block relative to theposition of the first initial reference block, and the second positionoffset (delta1x, delta1y) represents an offset of the position of thesecond reference block relative to the position of the second initialreference block. In particular, for positions of each pair of first andsecond reference blocks, the direction of the first offset is oppositeto the direction of the second offset, and the magnitude of the firstoffset is the same as the magnitude of the second offset, and the firstoffset and the second offset are respectively associated with therespective first reference block and the respective second referenceblock of the pair.

In block 640, a cost comparison between each pair of the first andsecond reference blocks among a plurality of first reference blocks inthe first reference picture and a plurality of second reference blocksin the second reference picture is performed. A cost comparison betweenthe first and second initial reference blocks may be also performed. Thefirst reference blocks may be, for example, the various reference blockssurrounding the first initial reference block 406 in the first referencepicture 408. The cost comparison is used to determine a first refinedmotion vector (e.g., MV0′) and a second refined motion vector (e.g.,MV1′).

The second reference blocks may be, for example, the various referenceblocks surrounding the second initial reference block 410 in the secondreference picture 412. Alternatively, positions of a pair of referenceblocks from the positions of the N pairs of reference blocks isdetermined as a position of the first refined reference block and aposition of the second refined reference block based on the matchingcost criterion. It can be understood that the N pairs of referenceblocks may include a pair of the first and second initial referenceblocks.

In block 642, a first refined reference block (e.g., refined referenceblock 416) in the first reference picture is selected based on the firstrefined motion vector and a second refined reference block (e.g.,refined reference block 418) in the second reference picture is selectedbased on the second refined motion vector. Alternatively, the firstrefined reference block is determined in the first reference picturebased on the position of the first refined reference block and thesecond refined reference block is determined in the second referencepicture based on the position of the second refined reference block.

In block 644, a prediction block (e.g., prediction block 420) isdetermined based on the first refined reference block and the secondrefined reference block.

In block 646, an image generated using the prediction block is displayedon the display of an electronic device.

Those skilled in the art will recognize that many solutions may beapplied to perform a decoder-side motion vector refinement (DMVR)procedure, and the present invention is not limited to the previouslyshown example processes.

Based on the above, the present disclosure allow conditionallyrestricting (such as enable or disable) decoder-side motion vectorrefinement based on the temporal distances to the two references used byeach CU, thus it improves the coding efficiency by not applying therefinement when the underlying assumption of equal and opposite deltamotion vectors that the refinement assumes is unlikely to be true.

Based on the above, the present disclosure also provides the granularityto disable refinement for all access units unconditionally, disablerefinement for certain access units conditionally, enable unconditionalrefinement at an access unit level, or enable conditional refinement atan access unit based on the temporal distances to the references fromthe current picture used by a coding unit within that access unit.

Based on the above, the present disclosure also offers the advantage ofdisabling the refinements performed on the decoder side for mergeindices that are unlikely to improve the compression gains.

Also, based on the above, the present disclosure restricts therefinements to only two equal-distance references can have advantage ofless cache pollution by the other references on the decoder side.

Based on the above, the present disclosure allows normative disabling ofbilateral matching based decoder-side motion vector refinement at the CUlevel whenever the temporal distances to the two references are notsubstantially equal in magnitude and opposite in direction. Inparticular, this disabling is applied when the refinement process doesnot use temporal distances to scale the delta motion vectors.

Based on the above, the present disclosure adds flags at the sequenceand picture parameter set levels to allow the temporal distance basedcheck at the CU level to be performed only when indicated by these flagsso that encoder has the option of signaling the appropriate flag valuesbased on factors such as GOP structure and coding gains seen.

Following is an explanation of the applications of the encoding methodas well as the decoding method as shown in the above-mentionedembodiments, and a system using them.

FIG. 11 is a block diagram showing a content supply system 3100 forrealizing content distribution service. This content supply system 3100includes capture device 3102, terminal device 3106, and optionallyincludes display 3126. The capture device 3102 communicates with theterminal device 3106 over communication link 3104. The communicationlink may include the communication channel 13 described above. Thecommunication link 3104 includes but not limited to WIFI, Ethernet,Cable, wireless (3G/4G/5G), USB, or any kind of combination thereof, orthe like.

The capture device 3102 generates data, and may encode the data by theencoding method as shown in the above embodiments. Alternatively, thecapture device 3102 may distribute the data to a streaming server (notshown in the Figures), and the server encodes the data and transmits theencoded data to the terminal device 3106. The capture device 3102includes but not limited to camera, smart phone or Pad, computer orlaptop, video conference system, PDA, vehicle mounted device, or acombination of any of them, or the like. For example, the capture device3102 may include the source device 12 as described above. When the dataincludes video, the video encoder 20 included in the capture device 3102may actually perform video encoding processing. When the data includesaudio (i.e., voice), an audio encoder included in the capture device3102 may actually perform audio encoding processing. For some practicalscenarios, the capture device 3102 distributes the encoded video andaudio data by multiplexing them together. For other practical scenarios,for example in the video conference system, the encoded audio data andthe encoded video data are not multiplexed. Capture device 3102distributes the encoded audio data and the encoded video data to theterminal device 3106 separately.

In the content supply system 3100, the terminal device 310 receives andreproduces the encoded data. The terminal device 3106 could be a devicewith data receiving and recovering capability, such as smart phone orPad 3108, computer or laptop 3110, network video recorder (NVR)/digitalvideo recorder (DVR) 3112, TV 3114, set top box (STB) 3116, videoconference system 3118, video surveillance system 3120, personal digitalassistant (PDA) 3122, vehicle mounted device 3124, or a combination ofany of them, or the like capable of decoding the above-mentioned encodeddata. For example, the terminal device 3106 may include the destinationdevice 14 as described above. When the encoded data includes video, thevideo decoder 30 included in the terminal device is prioritized toperform video decoding. When the encoded data includes audio, an audiodecoder included in the terminal device is prioritized to perform audiodecoding processing.

For a terminal device with its display, for example, smart phone or Pad3108, computer or laptop 3110, network video recorder (NVR)/digitalvideo recorder (DVR) 3112, TV 3114, personal digital assistant (PDA)3122, or vehicle mounted device 3124, the terminal device can feed thedecoded data to its display. For a terminal device equipped with nodisplay, for example, STB 3116, video conference system 3118, or videosurveillance system 3120, an external display 3126 is contacted thereinto receive and show the decoded data.

When each device in this system performs encoding or decoding, thepicture encoding device or the picture decoding device, as shown in theabove-mentioned embodiments, can be used.

FIG. 12 is a diagram showing a structure of an example of the terminaldevice 3106. After the terminal device 3106 receives stream from thecapture device 3102, the protocol proceeding unit 3202 analyzes thetransmission protocol of the stream. The protocol includes but notlimited to Real Time Streaming Protocol (RTSP), Hyper Text TransferProtocol (HTTP), HTTP Live streaming protocol (HLS), MPEG-DASH,Real-time Transport protocol (RTP), Real Time Messaging Protocol (RTMP),or any kind of combination thereof, or the like.

After the protocol proceeding unit 3202 processes the stream, streamfile is generated. The file is outputted to a demultiplexing unit 3204.The demultiplexing unit 3204 can separate the multiplexed data into theencoded audio data and the encoded video data. As described above, forsome practical scenarios, for example in the video conference system,the encoded audio data and the encoded video data are not multiplexed.In this situation, the encoded data is transmitted to video decoder 3206and audio decoder 3208 without through the demultiplexing unit 3204.

Via the demultiplexing processing, video elementary stream (ES), audioES, and optionally subtitle are generated. The video decoder 3206, whichincludes the video decoder 30 as explained in the above mentionedembodiments, decodes the video ES by the decoding method as shown in theabove-mentioned embodiments to generate video frame, and feeds this datato the synchronous unit 3212. The audio decoder 3208, decodes the audioES to generate audio frame, and feeds this data to the synchronous unit3212. Alternatively, the video frame may store in a buffer (not shown inFIG. Y) before feeding it to the synchronous unit 3212. Similarly, theaudio frame may store in a buffer (not shown in FIG. Y) before feedingit to the synchronous unit 3212.

The synchronous unit 3212 synchronizes the video frame and the audioframe, and supplies the video/audio to a video/audio display 3214. Forexample, the synchronous unit 3212 synchronizes the presentation of thevideo and audio information. Information may code in the syntax usingtime stamps concerning the presentation of coded audio and visual dataand time stamps concerning the delivery of the data stream itself.

If subtitle is included in the stream, the subtitle decoder 3210 decodesthe subtitle, and synchronizes it with the video frame and the audioframe, and supplies the video/audio/subtitle to a video/audio/subtitledisplay 3216.

The present invention is not limited to the above-mentioned system, andeither the picture encoding device or the picture decoding device in theabove-mentioned embodiments can be incorporated into other system, forexample, a car system.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over as oneor more instructions or code on a computer-readable medium and executedby a hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transitory media, but areinstead directed to non-transitory, tangible storage media. Disk anddisc, as used herein, includes compact disc (CD), laser disc, opticaldisc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wheredisks usually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated hardware and/or software modules configured for encoding anddecoding, or incorporated in a combined codec. Also, the techniquescould be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a codec hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

What is claimed is:
 1. A method for inter-prediction of a current imageblock in a current picture of a video, the method comprising:determining whether a first temporal distance is equal to a secondtemporal distance, wherein the first temporal distance is represented interms of a difference between a picture order count value of the currentpicture and a picture order count value of a first reference image; andthe second temporal distance is represented in terms of a differencebetween a picture order count value of a second reference image and thepicture order count value of the current picture; and performing motionvector refinement (DMVR) procedure to determine a prediction block ofthe current image block, when a plurality of conditions are satisfied,wherein one of the plurality of conditions is that it is determined thatthe first temporal distance is equal to the second temporal distance. 2.The method according to claim 1, further comprising: performing motioncompensation using a first initial motion vector and a second initialmotion vector to determine a prediction block of the current imageblock, when it is determined that the first temporal distance is notequal to the second temporal distance.
 3. The method according to claim1, the step of performing motion vector refinement (DMVR) procedure,comprises: determining a pair of best-matching blocks pointed to by bestmotion vectors from a plurality of pairs of reference blocks, whereinsaid pair of reference blocks includes a first reference block that isof the first reference picture and that is determined based on a firstinitial motion vector and a second reference block that is of the secondreference picture and that is determined based on a second initialmotion vector; wherein the best motion vectors include a first refinedmotion vector and a second refined motion vector.
 4. The methodaccording to claim 2, wherein initial motion information of the currentimage block comprises the first initial motion vector, a first referenceindex, the second initial motion vector and a second reference index,wherein the first reference index indicates the first reference picture,and the second reference index indicates the second reference picture.5. A method for inter-prediction of a current image block in a currentpicture of a video, the method comprising: determining whether a firsttemporal distance is equal to a second temporal distance, wherein thefirst temporal distance is represented in terms of a difference betweena picture order count value of the current picture and a picture ordercount value of a first reference image; and the second temporal distanceis represented in terms of a difference between a picture order countvalue of a second reference image and the picture order count value ofthe current picture; performing no motion vector refinement (DMVR)procedure on the current image block, when it is determined that thefirst temporal distance is not equal to the second temporal distance. 6.A coding device, comprising: a memory storing instructions; and aprocessor coupled to the memory, the processor configured to execute theinstructions stored in the memory to cause the processor to: determinewhether a first temporal distance is equal to a second temporaldistance, wherein the first temporal distance is represented in terms ofa difference between a picture order count value of a current pictureand a picture order count value of a first reference image; and thesecond temporal distance is represented in terms of a difference betweena picture order count value of a second reference image and the pictureorder count value of the current picture; and perform motion vectorrefinement (DMVR) procedure to determine a prediction block of thecurrent image block, when a plurality of conditions are satisfied,wherein one of the plurality of conditions is that it is determined thatthe first temporal distance is equal to the second temporal distance. 7.The device according to claim 6, wherein the processor is furtherconfigured to: perform motion compensation using a first initial motionvector and a second initial motion vector to determine a predictionblock of the current image block, when it is determined that the firsttemporal distance is not equal to the second temporal distance.
 8. Thedevice according to claim 6, wherein for the performing motion vectorrefinement (DMVR) procedure, the processor is configured to: determine apair of best-matching blocks pointed to by best motion vectors from aplurality of pairs of reference blocks, wherein said pair of referenceblocks includes a first reference block that is of the first referencepicture and that is determined based on a first initial motion vectorand a second reference block that is of the second reference picture andthat is determined based on a second initial motion vector; wherein thebest motion vectors include a first refined motion vector and a secondrefined motion vector.
 9. The device according to claim 7, wherein theprocessor is further configured to: obtain initial motion information ofthe current image block in the current picture, wherein the initialmotion information comprises the first initial motion vector, a firstreference index, the second initial motion vector and a second referenceindex, wherein the first reference index indicates the first referencepicture, and the second reference index indicates the second referencepicture.
 10. The device according to claim 6, wherein the codingapparatus is an encoding apparatus, wherein the processor is furtherconfigured to generate a bitstream including a residual between thecurrent image block and the prediction block, and an index forindicating initial motion information.
 11. The device according to claim6, wherein the coding apparatus is a decoding apparatus, wherein theprocessor is further configured to parse from a bitstream an index forindicating an initial motion information and a residual between acurrent image block and a prediction block of the current image block;and to reconstruct the current image block based on the residual and theprediction block.
 12. A coding device, comprising: a memory storinginstructions; and a processor coupled to the memory, the processorconfigured to execute the instructions stored in the memory to cause theprocessor to: determine whether a first temporal distance is equal to asecond temporal distance, wherein the first temporal distance isrepresented in terms of a difference between a picture order count valueof a current picture and a picture order count value of a firstreference image; and the second temporal distance is represented interms of a difference between a picture order count value of a secondreference image and the picture order count value of the currentpicture; and perform no motion vector refinement (DMVR) procedure on thecurrent image block, when it is determined that the first temporaldistance is not equal to the second temporal distance.
 13. An encodedbitstream for the video signal by including a plurality of syntaxelements, wherein the plurality of syntax elements comprises a firstsyntax element, wherein a value of the first syntax element indicateswhether a decoder-side motion vector refinement (DMVR) procedure isenabled or not; and wherein a plurality of steps are conditionallyperformed at least based on the value of the first syntax element, andthe steps comprising: determining whether a first temporal distance isequal to a second temporal distance, wherein the first temporal distanceis represented in terms of a difference between a picture order countvalue of the current picture and a picture order count value of a firstreference image; and the second temporal distance is represented interms of a difference between a picture order count value of a secondreference image and the picture order count value of the currentpicture; and performing motion vector refinement (DMVR) procedure todetermine a prediction block of the current image block, when aplurality of conditions are satisfied, wherein one of the plurality ofconditions is that it is determined that the first temporal distance isequal to the second temporal distance.
 14. The bitstream according toclaim 13, wherein the syntax element is signaled at any one of asequence parameter set (SPS) level, a picture parameter set (PPS) level,a slice header, coding tree unit (CTU) syntax, or coding unit (CU)syntax.
 15. An encoded bitstream for the video signal by including aplurality of syntax elements, wherein the plurality of syntax elementscomprises a first syntax element, wherein a value of the first syntaxelement indicates whether a decoder-side motion vector refinement (DMVR)procedure is enabled or not; and wherein a plurality of steps areconditionally performed at least based on the value of the first syntaxelement, and the steps comprising: determining whether a first temporaldistance is equal to a second temporal distance, wherein the firsttemporal distance is represented in terms of a difference between apicture order count value of the current picture and a picture ordercount value of a first reference image; and the second temporal distanceis represented in terms of a difference between a picture order countvalue of a second reference image and the picture order count value ofthe current picture; performing no motion vector refinement (DMVR)procedure on the current image block, when it is determined that thefirst temporal distance is not equal to the second temporal distance.16. The bitstream according to claim 15, wherein the syntax element issignaled at any one of a sequence parameter set (SPS) level, a pictureparameter set (PPS) level, a slice header, coding tree unit (CTU)syntax, or coding unit (CU) syntax.
 17. A computer-readable mediumstoring computer-readable instructions which when executed on aprocessor perform the steps comprising: determining whether a firsttemporal distance is equal to a second temporal distance, wherein thefirst temporal distance is represented in terms of a difference betweena picture order count value of the current picture and a picture ordercount value of a first reference image; and the second temporal distanceis represented in terms of a difference between a picture order countvalue of a second reference image and the picture order count value ofthe current picture; and performing motion vector refinement (DMVR)procedure to determine a prediction block of the current image block,when a plurality of conditions are satisfied, wherein one of theplurality of conditions is that it is determined that the first temporaldistance is equal to the second temporal distance.
 18. Thecomputer-readable medium according to claim 17, further comprising:performing motion compensation using a first initial motion vector and asecond initial motion vector to determine a prediction block of thecurrent image block, when it is determined that the first temporaldistance is not equal to the second temporal distance.
 19. Thecomputer-readable medium according to claim 18, wherein initial motioninformation of the current image block comprises the first initialmotion vector, a first reference index, the second initial motion vectorand a second reference index, wherein the first reference indexindicates the first reference picture, and the second reference indexindicates the second reference picture.
 20. A computer-readable mediumstoring computer-readable instructions which when executed on aprocessor perform the steps comprising: determining whether a firsttemporal distance is equal to a second temporal distance, wherein thefirst temporal distance is represented in terms of a difference betweena picture order count value of the current picture and a picture ordercount value of a first reference image; and the second temporal distanceis represented in terms of a difference between a picture order countvalue of a second reference image and the picture order count value ofthe current picture; performing no motion vector refinement (DMVR)procedure on the current image block, when it is determined that thefirst temporal distance is not equal to the second temporal distance.